theirix opened a new pull request, #23586:
URL: https://github.com/apache/datafusion/pull/23586
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases. You can
link an issue to this PR using the GitHub syntax. For example `Closes #123`
indicates that this PR will close issue #123.
-->
- Closes #.
## Rationale for this change
Performance optimisation in `lpad`/`rpad` UDF. Some codepaths do not
preallocate memory for the output string builder. Estimate capacity at lower
bound - works good for ASCII, slightly pessimistic for UTF8.
## What changes are included in this PR?
- Preallocate memory via `GenericStringBuilder::with_capacity`
## Are these changes tested?
- Bench shows significant improvement up to 65%
- Unit tests passed
- SLT passed
<details>
Running benches/pad.rs (target/release/deps/pad-032beeb29df3bbda)
Gnuplot not found, using plotters backend
Benchmarking lpad size=1024/lpad utf8 [size=1024, str_len=5, target=20]:
Collecting 10 samples in estimated 10.000 s (191k iter
lpad size=1024/lpad utf8 [size=1024, str_len=5, target=20]
time: [52.148 µs 52.852 µs 54.014 µs]
change: [−7.0729% −4.8343% −2.1990%] (p = 0.00 <
0.05)
Performance has improved.
Found 1 outliers among 10 measurements (10.00%)
1 (10.00%) high severe
Benchmarking lpad size=1024/lpad stringview [size=1024, str_len=5,
target=20]: Collecting 10 samples in estimated 10.000 s (188
lpad size=1024/lpad stringview [size=1024, str_len=5, target=20]
time: [52.554 µs 52.828 µs 53.121 µs]
change: [−43.495% −32.606% −20.991%] (p = 0.00 <
0.05)
Performance has improved.
Benchmarking lpad size=1024/lpad utf8 [size=1024, str_len=20, target=50]:
Collecting 10 samples in estimated 10.000 s (210k ite
lpad size=1024/lpad utf8 [size=1024, str_len=20, target=50]
time: [47.054 µs 47.217 µs 47.390 µs]
change: [−26.260% −23.376% −20.517%] (p = 0.00 <
0.05)
Performance has improved.
Benchmarking lpad size=1024/lpad stringview [size=1024, str_len=20,
target=50]: Collecting 10 samples in estimated 10.000 s (21
lpad size=1024/lpad stringview [size=1024, str_len=20, target=50]
time: [46.432 µs 46.612 µs 46.825 µs]
change: [−42.615% −31.873% −19.808%] (p = 0.00 <
0.05)
Performance has improved.
Found 3 outliers among 10 measurements (30.00%)
2 (20.00%) low mild
1 (10.00%) high severe
Benchmarking lpad size=1024/lpad utf8 unicode [size=1024, target=20]:
Collecting 10 samples in estimated 10.000 s (132k iterati
lpad size=1024/lpad utf8 unicode [size=1024, target=20]
time: [74.599 µs 74.871 µs 75.219 µs]
change: [−58.655% −49.993% −38.261%] (p = 0.00 <
0.05)
Performance has improved.
Found 1 outliers among 10 measurements (10.00%)
1 (10.00%) high mild
Benchmarking lpad size=1024/lpad stringview unicode [size=1024, target=20]:
Collecting 10 samples in estimated 10.001 s (131k i
lpad size=1024/lpad stringview unicode [size=1024, target=20]
time: [75.527 µs 76.618 µs 77.937 µs]
change: [−20.843% −17.960% −14.857%] (p = 0.00 <
0.05)
Performance has improved.
Benchmarking lpad size=1024/lpad utf8 scalar [size=1024, str_len=5,
target=20, fill='x']: Collecting 10 samples in estimated 10
lpad size=1024/lpad utf8 scalar [size=1024, str_len=5, target=20, fill='x']
time: [21.150 µs 21.278 µs 21.408 µs]
change: [−58.885% −46.240% −27.666%] (p = 0.00 <
0.05)
Performance has improved.
Benchmarking lpad size=1024/lpad stringview scalar [size=1024, str_len=5,
target=20, fill='x']: Collecting 10 samples in estima
lpad size=1024/lpad stringview scalar [size=1024, str_len=5, target=20,
fill='x']
time: [27.427 µs 27.616 µs 27.830 µs]
change: [−53.716% −43.022% −28.728%] (p = 0.00 <
0.05)
Performance has improved.
Benchmarking lpad size=1024/lpad utf8 scalar unicode [size=1024, str_len=5,
target=20, fill='é']: Collecting 10 samples in esti
lpad size=1024/lpad utf8 scalar unicode [size=1024, str_len=5, target=20,
fill='é']
time: [28.300 µs 28.484 µs 28.665 µs]
change: [−18.634% −16.104% −13.313%] (p = 0.00 <
0.05)
Performance has improved.
Benchmarking lpad size=1024/lpad utf8 scalar truncate [size=1024,
str_len=20, target=5, fill='é']: Collecting 10 samples in est
lpad size=1024/lpad utf8 scalar truncate [size=1024, str_len=20, target=5,
fill='é']
time: [23.336 µs 23.664 µs 24.025 µs]
change: [−40.196% −30.077% −19.258%] (p = 0.00 <
0.05)
Performance has improved.
Benchmarking lpad size=4096/lpad utf8 [size=4096, str_len=5, target=20]:
Collecting 10 samples in estimated 10.002 s (44k itera
lpad size=4096/lpad utf8 [size=4096, str_len=5, target=20]
time: [223.20 µs 224.76 µs 226.39 µs]
change: [−65.127% −61.080% −56.550%] (p = 0.00 <
0.05)
Performance has improved.
Benchmarking lpad size=4096/lpad stringview [size=4096, str_len=5,
target=20]: Collecting 10 samples in estimated 10.001 s (44k
lpad size=4096/lpad stringview [size=4096, str_len=5, target=20]
time: [225.52 µs 227.68 µs 229.85 µs]
change: [−27.889% −25.179% −22.048%] (p = 0.00 <
0.05)
Performance has improved.
Benchmarking lpad size=4096/lpad utf8 [size=4096, str_len=20, target=50]:
Collecting 10 samples in estimated 10.002 s (48k iter
lpad size=4096/lpad utf8 [size=4096, str_len=20, target=50]
time: [205.64 µs 207.16 µs 208.73 µs]
change: [−36.391% −33.620% −30.436%] (p = 0.00 <
0.05)
Performance has improved.
Benchmarking lpad size=4096/lpad stringview [size=4096, str_len=20,
target=50]: Collecting 10 samples in estimated 10.001 s (49
lpad size=4096/lpad stringview [size=4096, str_len=20, target=50]
time: [205.64 µs 211.63 µs 220.12 µs]
change: [−61.201% −52.916% −42.380%] (p = 0.00 <
0.05)
Performance has improved.
Found 1 outliers among 10 measurements (10.00%)
1 (10.00%) high severe
Benchmarking lpad size=4096/lpad utf8 unicode [size=4096, target=20]:
Collecting 10 samples in estimated 10.001 s (32k iteratio
lpad size=4096/lpad utf8 unicode [size=4096, target=20]
time: [308.13 µs 311.94 µs 316.17 µs]
change: [−26.606% −23.159% −19.590%] (p = 0.00 <
0.05)
Performance has improved.
Benchmarking lpad size=4096/lpad stringview unicode [size=4096, target=20]:
Collecting 10 samples in estimated 10.002 s (32k it
lpad size=4096/lpad stringview unicode [size=4096, target=20]
time: [315.30 µs 317.99 µs 321.15 µs]
change: [−44.713% −35.292% −25.783%] (p = 0.00 <
0.05)
Performance has improved.
Benchmarking lpad size=4096/lpad utf8 scalar [size=4096, str_len=5,
target=20, fill='x']: Collecting 10 samples in estimated 10
lpad size=4096/lpad utf8 scalar [size=4096, str_len=5, target=20, fill='x']
time: [80.809 µs 81.371 µs 81.976 µs]
change: [−37.978% −33.766% −29.556%] (p = 0.00 <
0.05)
Performance has improved.
Benchmarking lpad size=4096/lpad stringview scalar [size=4096, str_len=5,
target=20, fill='x']: Collecting 10 samples in estima
lpad size=4096/lpad stringview scalar [size=4096, str_len=5, target=20,
fill='x']
time: [106.70 µs 107.36 µs 108.04 µs]
change: [−59.140% −48.815% −33.758%] (p = 0.00 <
0.05)
Performance has improved.
Benchmarking lpad size=4096/lpad utf8 scalar unicode [size=4096, str_len=5,
target=20, fill='é']: Collecting 10 samples in esti
lpad size=4096/lpad utf8 scalar unicode [size=4096, str_len=5, target=20,
fill='é']
time: [103.68 µs 104.62 µs 105.65 µs]
change: [−55.113% −45.476% −33.018%] (p = 0.00 <
0.05)
Performance has improved.
Benchmarking lpad size=4096/lpad utf8 scalar truncate [size=4096,
str_len=20, target=5, fill='é']: Collecting 10 samples in est
lpad size=4096/lpad utf8 scalar truncate [size=4096, str_len=20, target=5,
fill='é']
time: [89.204 µs 89.915 µs 90.700 µs]
change: [−55.342% −42.291% −24.405%] (p = 0.00 <
0.05)
Performance has improved.
Found 2 outliers among 10 measurements (20.00%)
2 (20.00%) high mild
Benchmarking rpad size=1024/rpad utf8 [size=1024, str_len=5, target=20]:
Collecting 10 samples in estimated 10.000 s (180k iter
rpad size=1024/rpad utf8 [size=1024, str_len=5, target=20]
time: [54.901 µs 55.349 µs 55.853 µs]
change: [−37.177% −27.818% −19.121%] (p = 0.00 <
0.05)
Performance has improved.
Found 1 outliers among 10 measurements (10.00%)
1 (10.00%) high mild
Benchmarking rpad size=1024/rpad stringview [size=1024, str_len=5,
target=20]: Collecting 10 samples in estimated 10.000 s (183
rpad size=1024/rpad stringview [size=1024, str_len=5, target=20]
time: [53.203 µs 53.660 µs 54.176 µs]
change: [−61.294% −55.024% −46.285%] (p = 0.00 <
0.05)
Performance has improved.
Found 1 outliers among 10 measurements (10.00%)
1 (10.00%) high mild
Benchmarking rpad size=1024/rpad utf8 [size=1024, str_len=20, target=50]:
Collecting 10 samples in estimated 10.000 s (207k ite
rpad size=1024/rpad utf8 [size=1024, str_len=20, target=50]
time: [47.929 µs 48.691 µs 49.629 µs]
change: [−59.986% −56.927% −53.337%] (p = 0.00 <
0.05)
Performance has improved.
Found 2 outliers among 10 measurements (20.00%)
1 (10.00%) high mild
1 (10.00%) high severe
Benchmarking rpad size=1024/rpad stringview [size=1024, str_len=20,
target=50]: Collecting 10 samples in estimated 10.000 s (20
rpad size=1024/rpad stringview [size=1024, str_len=20, target=50]
time: [49.011 µs 49.469 µs 49.974 µs]
change: [−69.239% −64.995% −58.907%] (p = 0.00 <
0.05)
Performance has improved.
Benchmarking rpad size=1024/rpad utf8 unicode [size=1024, target=20]:
Collecting 10 samples in estimated 10.001 s (133k iterati
rpad size=1024/rpad utf8 unicode [size=1024, target=20]
time: [74.072 µs 74.680 µs 75.349 µs]
change: [−45.328% −34.674% −26.431%] (p = 0.00 <
0.05)
Performance has improved.
Found 1 outliers among 10 measurements (10.00%)
1 (10.00%) high mild
Benchmarking rpad size=1024/rpad stringview unicode [size=1024, target=20]:
Collecting 10 samples in estimated 10.000 s (133k i
rpad size=1024/rpad stringview unicode [size=1024, target=20]
time: [74.108 µs 74.794 µs 75.585 µs]
change: [−15.515% −10.211% −5.0875%] (p = 0.00 <
0.05)
Performance has improved.
Benchmarking rpad size=1024/rpad utf8 scalar [size=1024, str_len=5,
target=20, fill='x']: Collecting 10 samples in estimated 10
rpad size=1024/rpad utf8 scalar [size=1024, str_len=5, target=20, fill='x']
time: [21.632 µs 21.947 µs 22.316 µs]
change: [−36.830% −26.958% −14.540%] (p = 0.00 <
0.05)
Performance has improved.
Found 2 outliers among 10 measurements (20.00%)
2 (20.00%) high mild
Benchmarking rpad size=1024/rpad stringview scalar [size=1024, str_len=5,
target=20, fill='x']: Collecting 10 samples in estima
rpad size=1024/rpad stringview scalar [size=1024, str_len=5, target=20,
fill='x']
time: [27.751 µs 27.966 µs 28.195 µs]
change: [−36.301% −29.477% −22.101%] (p = 0.00 <
0.05)
Performance has improved.
Benchmarking rpad size=1024/rpad utf8 scalar unicode [size=1024, str_len=5,
target=20, fill='é']: Collecting 10 samples in esti
rpad size=1024/rpad utf8 scalar unicode [size=1024, str_len=5, target=20,
fill='é']
time: [27.495 µs 27.740 µs 28.027 µs]
change: [−14.694% −10.339% −6.2718%] (p = 0.00 <
0.05)
Performance has improved.
Found 1 outliers among 10 measurements (10.00%)
1 (10.00%) high mild
Benchmarking rpad size=1024/rpad utf8 scalar truncate [size=1024,
str_len=20, target=5, fill='é']: Collecting 10 samples in est
rpad size=1024/rpad utf8 scalar truncate [size=1024, str_len=20, target=5,
fill='é']
time: [22.598 µs 22.753 µs 22.992 µs]
change: [−25.113% −18.974% −12.648%] (p = 0.00 <
0.05)
Performance has improved.
Found 1 outliers among 10 measurements (10.00%)
1 (10.00%) high severe
Benchmarking rpad size=4096/rpad utf8 [size=4096, str_len=5, target=20]:
Collecting 10 samples in estimated 10.001 s (43k itera
rpad size=4096/rpad utf8 [size=4096, str_len=5, target=20]
time: [231.08 µs 232.61 µs 234.24 µs]
change: [−23.442% −18.719% −14.700%] (p = 0.00 <
0.05)
Performance has improved.
Benchmarking rpad size=4096/rpad stringview [size=4096, str_len=5,
target=20]: Collecting 10 samples in estimated 10.002 s (43k
rpad size=4096/rpad stringview [size=4096, str_len=5, target=20]
time: [225.59 µs 228.72 µs 232.57 µs]
change: [−5.5218% −3.7804% −1.8849%] (p = 0.00 <
0.05)
Performance has improved.
Found 2 outliers among 10 measurements (20.00%)
1 (10.00%) high mild
1 (10.00%) high severe
Benchmarking rpad size=4096/rpad utf8 [size=4096, str_len=20, target=50]:
Collecting 10 samples in estimated 10.001 s (48k iter
rpad size=4096/rpad utf8 [size=4096, str_len=20, target=50]
time: [209.11 µs 211.35 µs 213.85 µs]
change: [−5.8392% −4.4117% −2.8551%] (p = 0.00 <
0.05)
Performance has improved.
Found 2 outliers among 10 measurements (20.00%)
2 (20.00%) high mild
Benchmarking rpad size=4096/rpad stringview [size=4096, str_len=20,
target=50]: Collecting 10 samples in estimated 10.002 s (47
rpad size=4096/rpad stringview [size=4096, str_len=20, target=50]
time: [209.91 µs 211.87 µs 214.06 µs]
change: [−7.1173% −5.1876% −3.5064%] (p = 0.00 <
0.05)
Performance has improved.
Found 2 outliers among 10 measurements (20.00%)
2 (20.00%) high mild
Benchmarking rpad size=4096/rpad utf8 unicode [size=4096, target=20]:
Collecting 10 samples in estimated 10.003 s (29k iteratio
rpad size=4096/rpad utf8 unicode [size=4096, target=20]
time: [342.25 µs 344.74 µs 347.24 µs]
change: [+11.689% +12.876% +14.044%] (p = 0.00 <
0.05)
Performance has regressed.
Benchmarking rpad size=4096/rpad stringview unicode [size=4096, target=20]:
Collecting 10 samples in estimated 10.001 s (31k it
rpad size=4096/rpad stringview unicode [size=4096, target=20]
time: [314.81 µs 317.82 µs 321.01 µs]
change: [−1.2058% −0.0874% +1.0902%] (p = 0.89 >
0.05)
No change in performance detected.
Found 1 outliers among 10 measurements (10.00%)
1 (10.00%) high mild
Benchmarking rpad size=4096/rpad utf8 scalar [size=4096, str_len=5,
target=20, fill='x']: Collecting 10 samples in estimated 10
rpad size=4096/rpad utf8 scalar [size=4096, str_len=5, target=20, fill='x']
time: [81.538 µs 82.072 µs 82.749 µs]
change: [−6.9293% −5.8437% −4.8056%] (p = 0.00 <
0.05)
Performance has improved.
Found 1 outliers among 10 measurements (10.00%)
1 (10.00%) high mild
Benchmarking rpad size=4096/rpad stringview scalar [size=4096, str_len=5,
target=20, fill='x']: Collecting 10 samples in estima
rpad size=4096/rpad stringview scalar [size=4096, str_len=5, target=20,
fill='x']
time: [107.84 µs 108.79 µs 110.04 µs]
change: [−5.5539% −4.2839% −2.8920%] (p = 0.00 <
0.05)
Performance has improved.
Found 1 outliers among 10 measurements (10.00%)
1 (10.00%) high mild
Benchmarking rpad size=4096/rpad utf8 scalar unicode [size=4096, str_len=5,
target=20, fill='é']: Collecting 10 samples in esti
rpad size=4096/rpad utf8 scalar unicode [size=4096, str_len=5, target=20,
fill='é']
time: [148.04 µs 178.55 µs 208.57 µs]
change: [+40.729% +68.461% +93.445%] (p = 0.00 <
0.05)
Performance has regressed.
Benchmarking rpad size=4096/rpad utf8 scalar truncate [size=4096,
str_len=20, target=5, fill='é']: Collecting 10 samples in est
rpad size=4096/rpad utf8 scalar truncate [size=4096, str_len=20, target=5,
fill='é']
time: [102.17 µs 118.62 µs 142.26 µs]
change: [+13.952% +31.843% +59.302%] (p = 0.00 <
0.05)
Performance has regressed.
Found 2 outliers among 10 measurements (20.00%)
2 (20.00%) high severe
cargo bench --bench pad -- --baseline main-pad 1076.94s user 17.32s system
134% cpu 13:35.28 total
</details>
## Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
