[
https://issues.apache.org/jira/browse/ARROW-10117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17352945#comment-17352945
]
Weston Pace commented on ARROW-10117:
-------------------------------------
Another note is that a lot of academic literature (and techniques like Tokio's
thread pool) are willing to introduce a lot of complexity to maximize thread
pool performance. However, Arrow does not spawn millions of thread tasks
today. Tools like morsel-driven execution and fused operators allow us to run
with thousands of thread tasks per second. It's not clear to me how much
maintenance headaches we want to introduce to squeeze every last drop out of
the thread pool. At the moment I'm aiming to create the complex case first,
determine performance, implement a more naive work-stealing implementation, and
then re-benchmark so we can be more informed in the decision.
> [C++] Implement work-stealing scheduler / multiple queues in ThreadPool
> -----------------------------------------------------------------------
>
> Key: ARROW-10117
> URL: https://issues.apache.org/jira/browse/ARROW-10117
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C++
> Reporter: Wes McKinney
> Assignee: Weston Pace
> Priority: Major
>
> This involves a change from a single task queue shared amongst all threads to
> a per-thread task queue and the ability for idle threads to take tasks from
> other threads' queues (work stealing).
> As part of this, the task submission API would need to be evolved in some
> fashion to allow for tasks related to a particular workload to end up in the
> same task queue
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
