westonpace commented on pull request #11210:
URL: https://github.com/apache/arrow/pull/11210#issuecomment-938999032
This benchmark is very cool and we certainly need more benchmarks to
understand the overhead of ExecPlan. I'm not sure it explains the conbench
results however. The benchmark creates a Scan->Filter->Project plan and then
tests between serial & threaded.
However, the conbench result isn't a difference between has_executor=true
and has_executor=false.
The thing I most want to understand is the difference in performance between
"thread per batch" and "thread per operation". In other words, with your
change we now create 3 tasks for every batch in threaded mode. Before your
change we created 1 task for every batch.
Can you add a flag to your benchmark that only controls whether the filter &
project steps create a new task or use the serial runner. There would be two
modes:
Thread Per Batch Mode:
* ExecPlan::executor is set
* SourceNode creates tasks
* FilterNode and ProjectNode user serial runner
Thread Per Operation Mode:
* ExecPlan::executor is set (it is set in both)
* SourceNode creates tasks
* FilterNode and ProjectNode user parallel runner and create tasks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
