Rachelint commented on PR #15591: URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2863584067
The current benchmark results in my 16cores local: - target_partitions = 8 (amazing 1.71x faster) ``` // main Query 7 iteration 0 took 4256.6 ms and returned 10 rows Query 7 iteration 1 took 4265.5 ms and returned 10 rows Query 7 iteration 2 took 3822.5 ms and returned 10 rows Query 7 iteration 3 took 4136.2 ms and returned 10 rows Query 7 iteration 4 took 3851.7 ms and returned 10 rows Query 7 avg time: 4066.49 ms // blocked Query 7 iteration 0 took 2278.9 ms and returned 10 rows Query 7 iteration 1 took 2123.4 ms and returned 10 rows Query 7 iteration 2 took 2400.0 ms and returned 10 rows Query 7 iteration 3 took 2315.6 ms and returned 10 rows Query 7 iteration 4 took 2716.3 ms and returned 10 rows Query 7 avg time: 2366.87 ms ``` - target_partitions = 16(also 1.35x faster) ``` // main Query 7 iteration 0 took 1955.1 ms and returned 10 rows Query 7 iteration 1 took 2002.4 ms and returned 10 rows Query 7 iteration 2 took 1762.5 ms and returned 10 rows Query 7 iteration 3 took 1805.2 ms and returned 10 rows Query 7 iteration 4 took 1758.7 ms and returned 10 rows Query 7 avg time: 1856.78 ms // blocked Query 7 iteration 0 took 1392.8 ms and returned 10 rows Query 7 iteration 1 took 1357.3 ms and returned 10 rows Query 7 iteration 2 took 1378.4 ms and returned 10 rows Query 7 iteration 3 took 1376.6 ms and returned 10 rows Query 7 iteration 4 took 1367.1 ms and returned 10 rows Query 7 avg time: 1374.47 ms ``` - target_partitions = 32(still 1.10x faster) ``` // main Query 7 iteration 0 took 1471.3 ms and returned 10 rows Query 7 iteration 1 took 1350.5 ms and returned 10 rows Query 7 iteration 2 took 1361.4 ms and returned 10 rows Query 7 iteration 3 took 1393.6 ms and returned 10 rows Query 7 iteration 4 took 1427.4 ms and returned 10 rows Query 7 avg time: 1400.82 ms // blocked Query 7 iteration 0 took 1289.6 ms and returned 10 rows Query 7 iteration 1 took 1281.9 ms and returned 10 rows Query 7 iteration 2 took 1293.4 ms and returned 10 rows Query 7 iteration 3 took 1258.8 ms and returned 10 rows Query 7 iteration 4 took 1279.4 ms and returned 10 rows Query 7 avg time: 1280.62 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
