theirix commented on PR #23586:
URL: https://github.com/apache/datafusion/pull/23586#issuecomment-4984137575
I think the benchmark is quite noisy - probably an unseeded `rand::rng`. For
example, scalar-length benches named ".*scalar.*" (e.g. `rpad utf8 scalar`) do
not even touch changed code in `lpad_impl` - they go directly to the fast path
in `lpad_scalar_args`, but show both slowdown and speedup.
Array-length benches indeed measure the introduced change in the first
branch of `lpad_impl`. I guess it could not show a huge benefit on a short
array (5-20 items).
I can improve the bench to use a seeded generator and a larger array.
Remeasured with
```sh
git checkout main ; time cargo bench --bench pad -- --save-baseline
main-pad2 ; git checkout perf-pad-preallocate ; time cargo bench --bench pad --
--baseline main-pad2
```
:
```
Gnuplot not found, using plotters backend
Benchmarking lpad size=1024/lpad utf8 [size=1024, str_len=5, target=20]:
Collecting 10 samples in estimated 10.000 s (190k iter
lpad size=1024/lpad utf8 [size=1024, str_len=5, target=20]
time: [50.951 µs 51.296 µs 51.633 µs]
change: [+0.1946% +1.1167% +1.9898%] (p = 0.03 <
0.05)
Change within noise threshold.
Benchmarking lpad size=1024/lpad stringview [size=1024, str_len=5,
target=20]: Collecting 10 samples in estimated 10.000 s (198
lpad size=1024/lpad stringview [size=1024, str_len=5, target=20]
time: [50.185 µs 50.653 µs 51.140 µs]
change: [−1.5913% −0.4468% +0.7622%] (p = 0.49 >
0.05)
No change in performance detected.
Benchmarking lpad size=1024/lpad utf8 [size=1024, str_len=20, target=50]:
Collecting 10 samples in estimated 10.000 s (225k ite
lpad size=1024/lpad utf8 [size=1024, str_len=20, target=50]
time: [43.879 µs 44.197 µs 44.532 µs]
change: [−5.0874% −3.7927% −2.5465%] (p = 0.00 <
0.05)
Performance has improved.
Benchmarking lpad size=1024/lpad stringview [size=1024, str_len=20,
target=50]: Collecting 10 samples in estimated 10.000 s (21
lpad size=1024/lpad stringview [size=1024, str_len=20, target=50]
time: [45.068 µs 45.426 µs 45.758 µs]
change: [−5.5685% −4.6639% −3.7502%] (p = 0.00 <
0.05)
Performance has improved.
Benchmarking lpad size=1024/lpad utf8 unicode [size=1024, target=20]:
Collecting 10 samples in estimated 10.000 s (139k iterati
lpad size=1024/lpad utf8 unicode [size=1024, target=20]
time: [71.148 µs 71.630 µs 72.116 µs]
change: [−5.5971% −4.2385% −2.9885%] (p = 0.00 <
0.05)
Performance has improved.
Benchmarking lpad size=1024/lpad stringview unicode [size=1024, target=20]:
Collecting 10 samples in estimated 10.000 s (129k i
lpad size=1024/lpad stringview unicode [size=1024, target=20]
time: [76.803 µs 77.195 µs 77.639 µs]
change: [−1.3026% −0.3030% +0.7096%] (p = 0.58 >
0.05)
No change in performance detected.
Found 1 outliers among 10 measurements (10.00%)
1 (10.00%) high mild
Benchmarking lpad size=1024/lpad utf8 scalar [size=1024, str_len=5,
target=20, fill='x']: Collecting 10 samples in estimated 10
lpad size=1024/lpad utf8 scalar [size=1024, str_len=5, target=20, fill='x']
time: [20.451 µs 20.622 µs 20.800 µs]
change: [−7.4734% −6.2815% −5.1379%] (p = 0.00 <
0.05)
Performance has improved.
Benchmarking lpad size=1024/lpad stringview scalar [size=1024, str_len=5,
target=20, fill='x']: Collecting 10 samples in estima
lpad size=1024/lpad stringview scalar [size=1024, str_len=5, target=20,
fill='x']
time: [26.081 µs 26.312 µs 26.575 µs]
change: [−6.8776% −5.9423% −4.9372%] (p = 0.00 <
0.05)
Performance has improved.
Benchmarking lpad size=1024/lpad utf8 scalar unicode [size=1024, str_len=5,
target=20, fill='é']: Collecting 10 samples in esti
lpad size=1024/lpad utf8 scalar unicode [size=1024, str_len=5, target=20,
fill='é']
time: [27.541 µs 27.737 µs 27.930 µs]
change: [−4.0327% −2.5571% −1.0765%] (p = 0.01 <
0.05)
Performance has improved.
Benchmarking lpad size=1024/lpad utf8 scalar truncate [size=1024,
str_len=20, target=5, fill='é']: Collecting 10 samples in est
lpad size=1024/lpad utf8 scalar truncate [size=1024, str_len=20, target=5,
fill='é']
time: [22.537 µs 22.687 µs 22.834 µs]
change: [−7.5906% −6.3647% −5.1506%] (p = 0.00 <
0.05)
Performance has improved.
Benchmarking lpad size=4096/lpad utf8 [size=4096, str_len=5, target=20]:
Collecting 10 samples in estimated 10.002 s (43k itera
lpad size=4096/lpad utf8 [size=4096, str_len=5, target=20]
time: [222.76 µs 224.68 µs 226.85 µs]
change: [+1.4288% +2.8650% +4.3918%] (p = 0.00 <
0.05)
Performance has regressed.
Found 1 outliers among 10 measurements (10.00%)
1 (10.00%) high mild
Benchmarking lpad size=4096/lpad stringview [size=4096, str_len=5,
target=20]: Collecting 10 samples in estimated 10.001 s (46k
lpad size=4096/lpad stringview [size=4096, str_len=5, target=20]
time: [215.04 µs 216.95 µs 218.93 µs]
change: [−2.6256% −1.6363% −0.6260%] (p = 0.01 <
0.05)
Change within noise threshold.
Benchmarking lpad size=4096/lpad utf8 [size=4096, str_len=20, target=50]:
Collecting 10 samples in estimated 10.001 s (48k iter
lpad size=4096/lpad utf8 [size=4096, str_len=20, target=50]
time: [201.67 µs 202.82 µs 203.86 µs]
change: [−4.9848% −3.9805% −3.1575%] (p = 0.00 <
0.05)
Performance has improved.
Benchmarking lpad size=4096/lpad stringview [size=4096, str_len=20,
target=50]: Collecting 10 samples in estimated 10.002 s (49
lpad size=4096/lpad stringview [size=4096, str_len=20, target=50]
time: [197.42 µs 199.11 µs 201.08 µs]
change: [−8.8123% −7.6406% −6.4674%] (p = 0.00 <
0.05)
Performance has improved.
Found 1 outliers among 10 measurements (10.00%)
1 (10.00%) high mild
Benchmarking lpad size=4096/lpad utf8 unicode [size=4096, target=20]:
Collecting 10 samples in estimated 10.002 s (31k iteratio
lpad size=4096/lpad utf8 unicode [size=4096, target=20]
time: [300.50 µs 304.58 µs 308.45 µs]
change: [−3.9743% −2.1667% −0.6576%] (p = 0.02 <
0.05)
Change within noise threshold.
Benchmarking lpad size=4096/lpad stringview unicode [size=4096, target=20]:
Collecting 10 samples in estimated 10.000 s (31k it
lpad size=4096/lpad stringview unicode [size=4096, target=20]
time: [318.05 µs 325.55 µs 335.18 µs]
change: [−0.5558% +1.8560% +5.4023%] (p = 0.24 >
0.05)
No change in performance detected.
Found 1 outliers among 10 measurements (10.00%)
1 (10.00%) high mild
Benchmarking lpad size=4096/lpad utf8 scalar [size=4096, str_len=5,
target=20, fill='x']: Collecting 10 samples in estimated 10
lpad size=4096/lpad utf8 scalar [size=4096, str_len=5, target=20, fill='x']
time: [77.626 µs 78.838 µs 80.472 µs]
change: [−7.4475% −5.7844% −3.7876%] (p = 0.00 <
0.05)
Performance has improved.
Found 1 outliers among 10 measurements (10.00%)
1 (10.00%) high severe
Benchmarking lpad size=4096/lpad stringview scalar [size=4096, str_len=5,
target=20, fill='x']: Collecting 10 samples in estima
lpad size=4096/lpad stringview scalar [size=4096, str_len=5, target=20,
fill='x']
time: [101.52 µs 102.36 µs 103.24 µs]
change: [−6.6891% −5.3772% −4.1194%] (p = 0.00 <
0.05)
Performance has improved.
Benchmarking lpad size=4096/lpad utf8 scalar unicode [size=4096, str_len=5,
target=20, fill='é']: Collecting 10 samples in esti
lpad size=4096/lpad utf8 scalar unicode [size=4096, str_len=5, target=20,
fill='é']
time: [100.80 µs 101.54 µs 102.28 µs]
change: [−1.7351% −0.8442% +0.0923%] (p = 0.11 >
0.05)
No change in performance detected.
Benchmarking lpad size=4096/lpad utf8 scalar truncate [size=4096,
str_len=20, target=5, fill='é']: Collecting 10 samples in est
lpad size=4096/lpad utf8 scalar truncate [size=4096, str_len=20, target=5,
fill='é']
time: [85.114 µs 85.728 µs 86.430 µs]
change: [−8.1141% −7.0716% −5.9877%] (p = 0.00 <
0.05)
Performance has improved.
Found 1 outliers among 10 measurements (10.00%)
1 (10.00%) high mild
Benchmarking rpad size=1024/rpad utf8 [size=1024, str_len=5, target=20]:
Collecting 10 samples in estimated 10.001 s (182k iter
rpad size=1024/rpad utf8 [size=1024, str_len=5, target=20]
time: [53.693 µs 54.226 µs 54.787 µs]
change: [−1.4109% −0.1854% +1.0149%] (p = 0.77 >
0.05)
No change in performance detected.
Benchmarking rpad size=1024/rpad stringview [size=1024, str_len=5,
target=20]: Collecting 10 samples in estimated 10.000 s (193
rpad size=1024/rpad stringview [size=1024, str_len=5, target=20]
time: [50.586 µs 50.965 µs 51.368 µs]
change: [−6.5557% −5.4399% −4.2922%] (p = 0.00 <
0.05)
Performance has improved.
Benchmarking rpad size=1024/rpad utf8 [size=1024, str_len=20, target=50]:
Collecting 10 samples in estimated 10.000 s (212k ite
rpad size=1024/rpad utf8 [size=1024, str_len=20, target=50]
time: [46.114 µs 46.582 µs 47.059 µs]
change: [−0.2528% +1.2908% +2.7368%] (p = 0.12 >
0.05)
No change in performance detected.
Benchmarking rpad size=1024/rpad stringview [size=1024, str_len=20,
target=50]: Collecting 10 samples in estimated 10.000 s (20
rpad size=1024/rpad stringview [size=1024, str_len=20, target=50]
time: [47.812 µs 48.150 µs 48.509 µs]
change: [−4.0438% −2.9366% −1.7009%] (p = 0.00 <
0.05)
Performance has improved.
Benchmarking rpad size=1024/rpad utf8 unicode [size=1024, target=20]:
Collecting 10 samples in estimated 10.000 s (129k iterati
rpad size=1024/rpad utf8 unicode [size=1024, target=20]
time: [75.965 µs 76.664 µs 77.460 µs]
change: [+2.4628% +4.0885% +5.6688%] (p = 0.00 <
0.05)
Performance has regressed.
Benchmarking rpad size=1024/rpad stringview unicode [size=1024, target=20]:
Collecting 10 samples in estimated 10.000 s (130k i
rpad size=1024/rpad stringview unicode [size=1024, target=20]
time: [74.875 µs 75.514 µs 76.137 µs]
change: [−2.9222% −1.8287% −0.7415%] (p = 0.01 <
0.05)
Change within noise threshold.
Benchmarking rpad size=1024/rpad utf8 scalar [size=1024, str_len=5,
target=20, fill='x']: Collecting 10 samples in estimated 10
rpad size=1024/rpad utf8 scalar [size=1024, str_len=5, target=20, fill='x']
time: [21.391 µs 21.527 µs 21.674 µs]
change: [−4.4094% −3.5282% −2.6291%] (p = 0.00 <
0.05)
Performance has improved.
Benchmarking rpad size=1024/rpad stringview scalar [size=1024, str_len=5,
target=20, fill='x']: Collecting 10 samples in estima
rpad size=1024/rpad stringview scalar [size=1024, str_len=5, target=20,
fill='x']
time: [26.233 µs 26.461 µs 26.695 µs]
change: [−7.9990% −6.9383% −5.8495%] (p = 0.00 <
0.05)
Performance has improved.
Benchmarking rpad size=1024/rpad utf8 scalar unicode [size=1024, str_len=5,
target=20, fill='é']: Collecting 10 samples in esti
rpad size=1024/rpad utf8 scalar unicode [size=1024, str_len=5, target=20,
fill='é']
time: [29.309 µs 29.526 µs 29.763 µs]
change: [+2.7147% +3.8823% +5.0890%] (p = 0.00 <
0.05)
Performance has regressed.
Benchmarking rpad size=1024/rpad utf8 scalar truncate [size=1024,
str_len=20, target=5, fill='é']: Collecting 10 samples in est
rpad size=1024/rpad utf8 scalar truncate [size=1024, str_len=20, target=5,
fill='é']
time: [23.521 µs 23.757 µs 23.994 µs]
change: [−0.2927% +1.0385% +2.2395%] (p = 0.13 >
0.05)
No change in performance detected.
Benchmarking rpad size=4096/rpad utf8 [size=4096, str_len=5, target=20]:
Collecting 10 samples in estimated 10.001 s (42k itera
rpad size=4096/rpad utf8 [size=4096, str_len=5, target=20]
time: [231.70 µs 233.67 µs 235.90 µs]
change: [−1.7625% −0.5461% +0.7158%] (p = 0.41 >
0.05)
No change in performance detected.
Benchmarking rpad size=4096/rpad stringview [size=4096, str_len=5,
target=20]: Collecting 10 samples in estimated 10.002 s (45k
rpad size=4096/rpad stringview [size=4096, str_len=5, target=20]
time: [221.51 µs 223.33 µs 225.11 µs]
change: [−5.1227% −4.0799% −3.0239%] (p = 0.00 <
0.05)
Performance has improved.
Benchmarking rpad size=4096/rpad utf8 [size=4096, str_len=20, target=50]:
Collecting 10 samples in estimated 10.001 s (48k iter
rpad size=4096/rpad utf8 [size=4096, str_len=20, target=50]
time: [203.82 µs 205.31 µs 206.87 µs]
change: [−6.5032% −5.4654% −4.2901%] (p = 0.00 <
0.05)
Performance has improved.
Benchmarking rpad size=4096/rpad stringview [size=4096, str_len=20,
target=50]: Collecting 10 samples in estimated 10.001 s (47
rpad size=4096/rpad stringview [size=4096, str_len=20, target=50]
time: [206.84 µs 208.62 µs 210.40 µs]
change: [−5.2228% −4.3117% −3.4398%] (p = 0.00 <
0.05)
Performance has improved.
Benchmarking rpad size=4096/rpad utf8 unicode [size=4096, target=20]:
Collecting 10 samples in estimated 10.001 s (32k iteratio
rpad size=4096/rpad utf8 unicode [size=4096, target=20]
time: [307.57 µs 310.43 µs 313.50 µs]
change: [+1.8416% +3.1713% +4.5211%] (p = 0.00 <
0.05)
Performance has regressed.
Benchmarking rpad size=4096/rpad stringview unicode [size=4096, target=20]:
Collecting 10 samples in estimated 10.001 s (32k it
rpad size=4096/rpad stringview unicode [size=4096, target=20]
time: [310.70 µs 314.01 µs 317.62 µs]
change: [−1.0224% +0.0726% +1.4281%] (p = 0.91 >
0.05)
No change in performance detected.
Found 1 outliers among 10 measurements (10.00%)
1 (10.00%) high mild
Benchmarking rpad size=4096/rpad utf8 scalar [size=4096, str_len=5,
target=20, fill='x']: Collecting 10 samples in estimated 10
rpad size=4096/rpad utf8 scalar [size=4096, str_len=5, target=20, fill='x']
time: [80.940 µs 81.646 µs 82.328 µs]
change: [−4.9985% −3.8865% −2.7524%] (p = 0.00 <
0.05)
Performance has improved.
Benchmarking rpad size=4096/rpad stringview scalar [size=4096, str_len=5,
target=20, fill='x']: Collecting 10 samples in estima
rpad size=4096/rpad stringview scalar [size=4096, str_len=5, target=20,
fill='x']
time: [102.70 µs 103.46 µs 104.24 µs]
change: [−8.8161% −7.7952% −6.7599%] (p = 0.00 <
0.05)
Performance has improved.
Benchmarking rpad size=4096/rpad utf8 scalar unicode [size=4096, str_len=5,
target=20, fill='é']: Collecting 10 samples in esti
rpad size=4096/rpad utf8 scalar unicode [size=4096, str_len=5, target=20,
fill='é']
time: [108.27 µs 109.33 µs 110.39 µs]
change: [+0.6159% +2.8719% +5.0294%] (p = 0.03 <
0.05)
Change within noise threshold.
Benchmarking rpad size=4096/rpad utf8 scalar truncate [size=4096,
str_len=20, target=5, fill='é']: Collecting 10 samples in est
rpad size=4096/rpad utf8 scalar truncate [size=4096, str_len=20, target=5,
fill='é']
time: [89.067 µs 89.805 µs 90.618 µs]
change: [+0.1247% +1.2650% +2.4609%] (p = 0.06 >
0.05)
No change in performance detected.
cargo bench --bench pad -- --baseline main-pad2 1068.83s user 12.77s system
136% cpu 13:13.49 total
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
