Andy Grove created ARROW-9707:
---------------------------------
Summary: [Rust] [DataFusion] Re-implement threading model
Key: ARROW-9707
URL: https://issues.apache.org/jira/browse/ARROW-9707
Project: Apache Arrow
Issue Type: Sub-task
Components: Rust, Rust - DataFusion
Reporter: Andy Grove
Assignee: Andy Grove
Fix For: 2.0.0
The current threading model is very simple and does not scale. We currently use
1-2 dedicated threads per partition and they all run simultaneously, which is a
huge problem if you have more partitions than logical or physical cores.
This task is to re-implement the threading model so that query execution uses a
fixed (configurable) number of threads. Work will be broken down into stages
and tasks and each in-process executor (running on a dedicated thread) will
process its queue of tasks.
This process will be driven by a scheduler.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
