neilconway commented on PR #20588:
URL: https://github.com/apache/datafusion/pull/20588#issuecomment-3994664263
Here are the results on the Hetzner machine with 512 row chunks:
```
group base
target
----- ----
------
array_has_all/all_found_small_needle/10 4.73 6.5±0.03ms
? ?/sec 1.00 1377.3±7.28µs ? ?/sec
array_has_all/all_found_small_needle/100 1.50 15.5±0.05ms
? ?/sec 1.00 10.3±0.03ms ? ?/sec
array_has_all/all_found_small_needle/500 1.05 54.7±0.15ms
? ?/sec 1.00 52.2±1.56ms ? ?/sec
array_has_all/not_all_found/10 5.84 6.3±0.07ms
? ?/sec 1.00 1087.9±4.54µs ? ?/sec
array_has_all/not_all_found/100 1.60 14.3±0.33ms
? ?/sec 1.00 9.0±0.08ms ? ?/sec
array_has_all/not_all_found/500 1.10 49.0±0.13ms
? ?/sec 1.00 44.4±0.50ms ? ?/sec
array_has_all_strings/all_found/10 2.73 5.4±0.02ms
? ?/sec 1.00 1958.8±19.92µs ? ?/sec
array_has_all_strings/all_found/100 1.36 15.1±0.06ms
? ?/sec 1.00 11.1±0.08ms ? ?/sec
array_has_all_strings/all_found/500 1.13 60.6±1.65ms
? ?/sec 1.00 53.8±1.26ms ? ?/sec
array_has_all_strings/not_all_found/10 3.03 4.0±0.04ms
? ?/sec 1.00 1305.8±9.69µs ? ?/sec
array_has_all_strings/not_all_found/100 1.42 13.6±0.08ms
? ?/sec 1.00 9.5±0.05ms ? ?/sec
array_has_all_strings/not_all_found/500 1.14 69.7±0.27ms
? ?/sec 1.00 61.1±0.32ms ? ?/sec
array_has_any/no_match/10 3.23 7.3±0.04ms
? ?/sec 1.00 2.3±0.01ms ? ?/sec
array_has_any/no_match/100 1.22 22.9±0.10ms
? ?/sec 1.00 18.8±0.05ms ? ?/sec
array_has_any/no_match/500 1.00 92.3±0.24ms
? ?/sec 1.01 93.2±0.39ms ? ?/sec
array_has_any/scalar_no_match/10 1.00 2.2±0.02ms
? ?/sec 1.00 2.2±0.01ms ? ?/sec
array_has_any/scalar_no_match/100 1.00 20.8±0.17ms
? ?/sec 1.00 20.9±0.11ms ? ?/sec
array_has_any/scalar_no_match/500 1.00 136.6±1.66ms
? ?/sec 1.02 140.0±1.22ms ? ?/sec
array_has_any/scalar_some_match/10 1.00 1069.6±17.71µs
? ?/sec 1.01 1075.2±5.81µs ? ?/sec
array_has_any/scalar_some_match/100 1.00 11.0±0.08ms
? ?/sec 1.01 11.1±0.08ms ? ?/sec
array_has_any/scalar_some_match/500 1.00 84.8±0.51ms
? ?/sec 1.01 85.7±0.71ms ? ?/sec
array_has_any/some_match/10 5.06 6.4±0.04ms
? ?/sec 1.00 1257.1±4.21µs ? ?/sec
array_has_any/some_match/100 1.46 14.6±0.07ms
? ?/sec 1.00 10.0±0.19ms ? ?/sec
array_has_any/some_match/500 1.02 51.1±0.15ms
? ?/sec 1.00 50.0±0.33ms ? ?/sec
array_has_any_scalar/i64_no_match/1 1.00 375.2±4.65µs
? ?/sec 1.02 382.6±30.10µs ? ?/sec
array_has_any_scalar/i64_no_match/10 1.00 451.1±11.52µs
? ?/sec 1.03 464.7±10.41µs ? ?/sec
array_has_any_scalar/i64_no_match/100 1.01 638.5±27.58µs
? ?/sec 1.00 633.0±19.30µs ? ?/sec
array_has_any_scalar/i64_no_match/1000 1.00 543.6±11.89µs
? ?/sec 1.00 544.2±13.11µs ? ?/sec
array_has_any_scalar/string_no_match/1 1.00 249.8±1.86µs
? ?/sec 1.03 258.4±3.13µs ? ?/sec
array_has_any_scalar/string_no_match/10 1.00 419.9±8.88µs
? ?/sec 1.04 438.5±10.85µs ? ?/sec
array_has_any_scalar/string_no_match/100 1.00 550.3±23.91µs
? ?/sec 1.01 556.2±18.31µs ? ?/sec
array_has_any_scalar/string_no_match/1000 1.00 461.9±8.79µs
? ?/sec 1.01 465.6±7.16µs ? ?/sec
array_has_any_strings/no_match/10 2.04 5.0±0.03ms
? ?/sec 1.00 2.5±0.01ms ? ?/sec
array_has_any_strings/no_match/100 1.16 21.6±0.14ms
? ?/sec 1.00 18.7±0.09ms ? ?/sec
array_has_any_strings/no_match/500 1.01 129.2±0.40ms
? ?/sec 1.00 127.5±0.42ms ? ?/sec
array_has_any_strings/scalar_no_match/10 1.00 867.0±2.78µs
? ?/sec 1.07 926.9±9.70µs ? ?/sec
array_has_any_strings/scalar_no_match/100 1.00 7.4±0.02ms
? ?/sec 1.08 8.0±0.03ms ? ?/sec
array_has_any_strings/scalar_no_match/500 1.00 85.6±0.35ms
? ?/sec 1.07 92.0±0.37ms ? ?/sec
array_has_any_strings/scalar_some_match/10 1.00 764.9±12.75µs
? ?/sec 1.04 797.9±10.23µs ? ?/sec
array_has_any_strings/scalar_some_match/100 1.00 5.1±0.07ms
? ?/sec 1.06 5.4±0.05ms ? ?/sec
array_has_any_strings/scalar_some_match/500 1.00 17.3±0.10ms
? ?/sec 1.07 18.6±0.12ms ? ?/sec
array_has_any_strings/some_match/10 2.37 4.3±0.01ms
? ?/sec 1.00 1810.8±172.72µs ? ?/sec
array_has_any_strings/some_match/100 1.32 14.2±0.30ms
? ?/sec 1.00 10.7±0.26ms ? ?/sec
array_has_any_strings/some_match/500 1.04 52.4±0.16ms
? ?/sec 1.00 50.6±0.26ms ? ?/sec
array_has_i64/found/10 1.00 144.5±4.87µs
? ?/sec 1.03 148.4±4.69µs ? ?/sec
array_has_i64/found/100 1.00 629.1±59.20µs
? ?/sec 1.03 645.1±46.47µs ? ?/sec
array_has_i64/found/500 1.00 4.4±0.07ms
? ?/sec 1.04 4.6±0.22ms ? ?/sec
array_has_i64/not_found/10 1.04 69.4±0.56µs
? ?/sec 1.00 66.6±1.26µs ? ?/sec
array_has_i64/not_found/100 1.00 492.8±37.60µs
? ?/sec 1.00 491.9±38.98µs ? ?/sec
array_has_i64/not_found/500 1.00 4.3±0.09ms
? ?/sec 1.00 4.3±0.10ms ? ?/sec
array_has_strings/found/10 1.00 676.5±6.13µs
? ?/sec 1.01 686.6±7.26µs ? ?/sec
array_has_strings/found/100 1.00 2.7±0.05ms
? ?/sec 1.02 2.7±0.02ms ? ?/sec
array_has_strings/found/500 1.01 15.6±0.18ms
? ?/sec 1.00 15.5±0.22ms ? ?/sec
array_has_strings/not_found/10 1.02 152.4±1.15µs
? ?/sec 1.00 149.3±1.57µs ? ?/sec
array_has_strings/not_found/100 1.00 5.7±0.02ms
? ?/sec 1.01 5.8±0.01ms ? ?/sec
array_has_strings/not_found/500 1.00 16.2±0.04ms
? ?/sec 1.01 16.4±0.55ms ? ?/sec
```
I'm inclined to go with 512 row chunking: it seems that this reduces cache
pressure sufficiently, while doing half as many row-conversion calls as 256 row
chunking. I've updated the PR with that approach.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
