bkietz commented on pull request #8680: URL: https://github.com/apache/arrow/pull/8680#issuecomment-728333688
Added a benchmark to measure the cost of creating a Future in Executor::Submit instead of just using Executor::Spawn. Locally: ``` --------------------------------------------------------------------------------------------- Benchmark Time CPU Iterations --------------------------------------------------------------------------------------------- ThreadPoolSpawn/threads:4/task_cost:1000/real_time 201939036 ns 173389187 ns 6 967.19k items/s ThreadPoolSpawn/threads:8/task_cost:1000/real_time 294998893 ns 264699658 ns 2 662.082k items/s ThreadPoolSpawn/threads:4/task_cost:10000/real_time 36495616 ns 2546067 ns 19 535.194k items/s ThreadPoolSpawn/threads:8/task_cost:10000/real_time 20151885 ns 3711374 ns 30 969.251k items/s ThreadPoolSpawn/threads:4/task_cost:100000/real_time 36728060 ns 285424 ns 19 53.2046k items/s ThreadPoolSpawn/threads:8/task_cost:100000/real_time 23104241 ns 415593 ns 29 84.5776k items/s ThreadPoolSubmit/threads:4/task_cost:1000/real_time 20986410 ns 19332114 ns 33 465.377k items/s ThreadPoolSubmit/threads:8/task_cost:1000/real_time 19378929 ns 18328342 ns 36 503.98k items/s ThreadPoolSubmit/threads:4/task_cost:10000/real_time 2320937 ns 761124 ns 303 421.183k items/s ThreadPoolSubmit/threads:8/task_cost:10000/real_time 1727973 ns 1442820 ns 393 565.714k items/s ThreadPoolSubmit/threads:4/task_cost:100000/real_time 2085426 ns 112413 ns 329 47.2962k items/s ThreadPoolSubmit/threads:8/task_cost:100000/real_time 1279387 ns 104251 ns 514 77.0938k items/s ``` This shows there is significant overhead for small tasks and pools with fewer threads. Follow up: https://issues.apache.org/jira/browse/ARROW-10625 ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
