tustvold commented on PR #2677: URL: https://github.com/apache/arrow-datafusion/pull/2677#issuecomment-1168860442
Master uses spawn_blocking which effectively gives a single dedicated thread to file decode, this effectively gives it an unfair advantage when running benchmarks on an otherwise idle machine. However, compared against master with spawn_blocking disabled and with https://github.com/apache/arrow-rs/pull/1956 we get ``` tokio: select dict_10_optional from t time: [8.9050 ms 8.9173 ms 8.9295 ms] change: [-2.7950% -2.6195% -2.4525%] (p = 0.00 < 0.05) Performance has improved. Found 4 outliers among 100 measurements (4.00%) 3 (3.00%) low mild 1 (1.00%) high mild scheduled: select dict_10_optional from t time: [9.9143 ms 9.9508 ms 10.001 ms] change: [-4.0207% -3.6746% -3.2435%] (p = 0.00 < 0.05) Performance has improved. Found 7 outliers among 100 measurements (7.00%) 3 (3.00%) high mild 4 (4.00%) high severe tokio: select dict_100_optional from t time: [9.0259 ms 9.0440 ms 9.0644 ms] change: [-4.3203% -4.0547% -3.7845%] (p = 0.00 < 0.05) Performance has improved. Found 6 outliers among 100 measurements (6.00%) 2 (2.00%) high mild 4 (4.00%) high severe scheduled: select dict_100_optional from t time: [10.147 ms 10.159 ms 10.171 ms] change: [-3.7494% -3.5730% -3.3990%] (p = 0.00 < 0.05) Performance has improved. tokio: select dict_1000_optional from t time: [9.1436 ms 9.1600 ms 9.1782 ms] change: [-3.3663% -3.0933% -2.8203%] (p = 0.00 < 0.05) Performance has improved. Found 5 outliers among 100 measurements (5.00%) 1 (1.00%) high mild 4 (4.00%) high severe scheduled: select dict_1000_optional from t time: [10.196 ms 10.207 ms 10.220 ms] change: [-3.9349% -3.7611% -3.5989%] (p = 0.00 < 0.05) Performance has improved. Found 8 outliers among 100 measurements (8.00%) 7 (7.00%) high mild 1 (1.00%) high severe tokio: select count(*) from t where dict_10_required = 'prefix#0' time: [8.9137 ms 8.9359 ms 8.9579 ms] change: [-19.374% -18.929% -18.489%] (p = 0.00 < 0.05) Performance has improved. Found 3 outliers among 100 measurements (3.00%) 1 (1.00%) low mild 2 (2.00%) high mild scheduled: select count(*) from t where dict_10_required = 'prefix#0' time: [8.4053 ms 8.4312 ms 8.4574 ms] change: [-1.4720% -1.0417% -0.5966%] (p = 0.00 < 0.05) Change within noise threshold. tokio: select count(*) from t where dict_100_required = 'prefix#0' time: [7.7844 ms 7.8003 ms 7.8168 ms] change: [-12.353% -11.623% -10.882%] (p = 0.00 < 0.05) Performance has improved. Found 3 outliers among 100 measurements (3.00%) 3 (3.00%) high mild scheduled: select count(*) from t where dict_100_required = 'prefix#0' time: [7.2560 ms 7.3004 ms 7.3455 ms] change: [+0.6743% +1.2697% +1.9890%] (p = 0.00 < 0.05) Change within noise threshold. tokio: select count(*) from t where dict_1000_required = 'prefix#0' time: [7.3898 ms 7.4287 ms 7.4677 ms] change: [+0.4728% +1.1875% +1.9094%] (p = 0.00 < 0.05) Change within noise threshold. Found 1 outliers among 100 measurements (1.00%) 1 (1.00%) low mild scheduled: select count(*) from t where dict_1000_required = 'prefix#0' time: [6.6191 ms 6.6347 ms 6.6506 ms] change: [-2.3638% -1.9639% -1.5633%] (p = 0.00 < 0.05) Performance has improved. Found 4 outliers among 100 measurements (4.00%) 4 (4.00%) high mild tokio: select i64_optional from t where dict_10_required = 'prefix#2' and dict_1000_required = 'pref... time: [17.647 ms 17.673 ms 17.701 ms] change: [-4.0966% -3.7503% -3.4007%] (p = 0.00 < 0.05) Performance has improved. Found 8 outliers among 100 measurements (8.00%) 6 (6.00%) high mild 2 (2.00%) high severe scheduled: select i64_optional from t where dict_10_required = 'prefix#2' and dict_1000_required = '... time: [17.434 ms 17.460 ms 17.486 ms] change: [-3.2385% -2.9102% -2.5799%] (p = 0.00 < 0.05) Performance has improved. Found 1 outliers among 100 measurements (1.00%) 1 (1.00%) high mild tokio: select i64_required from t where dict_10_required = 'prefix#2' and dict_1000_required = 'pref... time: [16.511 ms 16.538 ms 16.566 ms] change: [-2.9112% -2.3987% -1.9441%] (p = 0.00 < 0.05) Performance has improved. scheduled: select i64_required from t where dict_10_required = 'prefix#2' and dict_1000_required = '... time: [16.202 ms 16.237 ms 16.271 ms] change: [-1.3293% -0.9066% -0.5069%] (p = 0.00 < 0.05) Change within noise threshold. Found 1 outliers among 100 measurements (1.00%) 1 (1.00%) low mild Benchmarking tokio: select string_optional from t where dict_10_required = 'prefix#1' and dict_1000_required = 'p...: Warming up for 3.0000 s Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 7.1s, or reduce sample count to 70. tokio: select string_optional from t where dict_10_required = 'prefix#1' and dict_1000_required = 'p... time: [70.205 ms 70.289 ms 70.378 ms] change: [-4.6151% -4.2040% -3.7925%] (p = 0.00 < 0.05) Performance has improved. Found 2 outliers among 100 measurements (2.00%) 2 (2.00%) high mild Benchmarking scheduled: select string_optional from t where dict_10_required = 'prefix#1' and dict_1000_required ...: Warming up for 3.0000 s Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 7.5s, or reduce sample count to 60. scheduled: select string_optional from t where dict_10_required = 'prefix#1' and dict_1000_required ... time: [74.526 ms 74.615 ms 74.699 ms] change: [-2.3271% -2.0266% -1.7270%] (p = 0.00 < 0.05) Performance has improved. Found 3 outliers among 100 measurements (3.00%) 3 (3.00%) low mild Benchmarking tokio: select string_required from t where dict_10_required = 'prefix#1' and dict_1000_required = 'p...: Warming up for 3.0000 s Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 12.0s, or reduce sample count to 40. tokio: select string_required from t where dict_10_required = 'prefix#1' and dict_1000_required = 'p... time: [119.38 ms 119.57 ms 119.78 ms] change: [+15.290% +15.988% +16.582%] (p = 0.00 < 0.05) Performance has regressed. Found 5 outliers among 100 measurements (5.00%) 5 (5.00%) high mild Benchmarking scheduled: select string_required from t where dict_10_required = 'prefix#1' and dict_1000_required ...: Warming up for 3.0000 s Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 12.4s, or reduce sample count to 40. scheduled: select string_required from t where dict_10_required = 'prefix#1' and dict_1000_required ... time: [123.68 ms 123.88 ms 124.11 ms] change: [+11.570% +12.168% +12.748%] (p = 0.00 < 0.05) Performance has regressed. Found 11 outliers among 100 measurements (11.00%) 6 (6.00%) high mild 5 (5.00%) high severe tokio: select distinct dict_10_required from t where dict_1000_optional is not NULL and i64_optional... time: [21.048 ms 21.074 ms 21.099 ms] change: [-2.9939% -2.6899% -2.3848%] (p = 0.00 < 0.05) Performance has improved. Found 2 outliers among 100 measurements (2.00%) 2 (2.00%) low mild scheduled: select distinct dict_10_required from t where dict_1000_optional is not NULL and i64_opti... time: [23.353 ms 23.404 ms 23.457 ms] change: [-2.2225% -1.9064% -1.5766%] (p = 0.00 < 0.05) Performance has improved. tokio: select distinct dict_10_required from t where dict_1000_optional is not NULL and i64_optional... #2 time: [21.421 ms 21.462 ms 21.505 ms] change: [-1.7021% -1.3476% -1.0027%] (p = 0.00 < 0.05) Performance has improved. Found 4 outliers among 100 measurements (4.00%) 4 (4.00%) high mild scheduled: select distinct dict_10_required from t where dict_1000_optional is not NULL and i64_opti... #2 time: [23.415 ms 23.485 ms 23.558 ms] change: [-1.6729% -1.2924% -0.9078%] (p = 0.00 < 0.05) Change within noise threshold. Found 2 outliers among 100 measurements (2.00%) 2 (2.00%) high mild tokio: select distinct dict_10_required from t where dict_1000_optional is not NULL and i64_required... time: [19.921 ms 19.967 ms 20.014 ms] change: [-1.2083% -0.7998% -0.4072%] (p = 0.00 < 0.05) Change within noise threshold. scheduled: select distinct dict_10_required from t where dict_1000_optional is not NULL and i64_requ... time: [21.143 ms 21.225 ms 21.310 ms] change: [-1.2061% -0.7093% -0.2452%] (p = 0.01 < 0.05) Change within noise threshold. tokio: select distinct dict_10_required from t where dict_1000_optional is not NULL and i64_required... #2 time: [20.369 ms 20.433 ms 20.495 ms] change: [+1.8375% +2.2683% +2.7020%] (p = 0.00 < 0.05) Performance has regressed. Found 7 outliers among 100 measurements (7.00%) 7 (7.00%) low mild scheduled: select distinct dict_10_required from t where dict_1000_optional is not NULL and i64_requ... #2 time: [21.510 ms 21.586 ms 21.663 ms] change: [+1.4549% +1.9070% +2.3468%] (p = 0.00 < 0.05) Performance has regressed. tokio: select dict_10_optional, count(*) from t group by dict_10_optional time: [23.850 ms 23.907 ms 23.963 ms] change: [-13.523% -13.268% -12.998%] (p = 0.00 < 0.05) Performance has improved. scheduled: select dict_10_optional, count(*) from t group by dict_10_optional time: [23.547 ms 23.598 ms 23.650 ms] change: [-1.3270% -1.0043% -0.6899%] (p = 0.00 < 0.05) Change within noise threshold. Found 1 outliers among 100 measurements (1.00%) 1 (1.00%) high mild Benchmarking tokio: select dict_10_optional, dict_100_optional, count(*) from t group by dict_10_optional, dict_1...: Warming up for 3.0000 s Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 5.3s, or reduce sample count to 90. tokio: select dict_10_optional, dict_100_optional, count(*) from t group by dict_10_optional, dict_1... time: [53.017 ms 53.120 ms 53.226 ms] change: [-16.476% -15.993% -15.511%] (p = 0.00 < 0.05) Performance has improved. Found 1 outliers among 100 measurements (1.00%) 1 (1.00%) high mild Benchmarking scheduled: select dict_10_optional, dict_100_optional, count(*) from t group by dict_10_optional, di...: Warming up for 3.0000 s Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 5.8s, or reduce sample count to 80. scheduled: select dict_10_optional, dict_100_optional, count(*) from t group by dict_10_optional, di... time: [56.802 ms 56.953 ms 57.102 ms] change: [-4.2172% -3.8167% -3.4177%] (p = 0.00 < 0.05) Performance has improved. Found 7 outliers among 100 measurements (7.00%) 4 (4.00%) low mild 3 (3.00%) high mild Benchmarking tokio: select dict_10_optional, dict_100_optional, MIN(f64_required), MAX(f64_required), AVG(f64_req...: Warming up for 3.0000 s Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 9.0s, or reduce sample count to 50. tokio: select dict_10_optional, dict_100_optional, MIN(f64_required), MAX(f64_required), AVG(f64_req... time: [90.751 ms 90.944 ms 91.147 ms] change: [-11.288% -10.904% -10.534%] (p = 0.00 < 0.05) Performance has improved. Found 2 outliers among 100 measurements (2.00%) 2 (2.00%) high mild Benchmarking scheduled: select dict_10_optional, dict_100_optional, MIN(f64_required), MAX(f64_required), AVG(f64...: Warming up for 3.0000 s Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 9.4s, or reduce sample count to 50. scheduled: select dict_10_optional, dict_100_optional, MIN(f64_required), MAX(f64_required), AVG(f64... time: [93.324 ms 93.439 ms 93.555 ms] change: [-3.1669% -2.9244% -2.6872%] (p = 0.00 < 0.05) Performance has improved. Found 1 outliers among 100 measurements (1.00%) 1 (1.00%) high mild Benchmarking tokio: select dict_10_optional, dict_100_optional, MIN(f64_optional), MAX(f64_optional), AVG(f64_opt...: Warming up for 3.0000 s Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 10.9s, or reduce sample count to 40. tokio: select dict_10_optional, dict_100_optional, MIN(f64_optional), MAX(f64_optional), AVG(f64_opt... time: [108.85 ms 108.97 ms 109.09 ms] change: [-14.437% -14.014% -13.595%] (p = 0.00 < 0.05) Performance has improved. Found 1 outliers among 100 measurements (1.00%) 1 (1.00%) high mild Benchmarking scheduled: select dict_10_optional, dict_100_optional, MIN(f64_optional), MAX(f64_optional), AVG(f64...: Warming up for 3.0000 s Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 11.2s, or reduce sample count to 40. scheduled: select dict_10_optional, dict_100_optional, MIN(f64_optional), MAX(f64_optional), AVG(f64... time: [112.08 ms 112.19 ms 112.29 ms] change: [-5.8059% -5.3435% -4.8943%] (p = 0.00 < 0.05) Performance has improved. Found 2 outliers among 100 measurements (2.00%) 2 (2.00%) low mild Benchmarking tokio: select dict_10_required, dict_100_required, MIN(f64_optional), MAX(f64_optional), AVG(f64_opt...: Warming up for 3.0000 s Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 10.2s, or reduce sample count to 40. tokio: select dict_10_required, dict_100_required, MIN(f64_optional), MAX(f64_optional), AVG(f64_opt... time: [102.02 ms 102.26 ms 102.62 ms] change: [-9.8760% -9.4604% -9.0112%] (p = 0.00 < 0.05) Performance has improved. Found 1 outliers among 100 measurements (1.00%) 1 (1.00%) high severe Benchmarking scheduled: select dict_10_required, dict_100_required, MIN(f64_optional), MAX(f64_optional), AVG(f64...: Warming up for 3.0000 s Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 10.2s, or reduce sample count to 40. scheduled: select dict_10_required, dict_100_required, MIN(f64_optional), MAX(f64_optional), AVG(f64... time: [102.21 ms 102.35 ms 102.50 ms] change: [-1.7080% -1.3686% -1.0517%] (p = 0.00 < 0.05) Performance has improved. ``` Which would suggest performance is comparable :tada: -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
