[ https://issues.apache.org/jira/browse/ARROW-8774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17105527#comment-17105527 ]
Wes McKinney commented on ARROW-8774: ------------------------------------- FYI we're also working on our thread scheduling and concurrent CPU/IO management APIs for C++. We might try to see if there are shared learnings, we can at least link issues on JIRA to point our design docs or work happening cc [~fsaintjacques] [~lidavidm] [~apitrou] > [Rust] [DataFusion] Improve threading model > ------------------------------------------- > > Key: ARROW-8774 > URL: https://issues.apache.org/jira/browse/ARROW-8774 > Project: Apache Arrow > Issue Type: Improvement > Components: Rust, Rust - DataFusion > Reporter: Andy Grove > Assignee: Andy Grove > Priority: Major > Fix For: 1.0.0 > > > DataFusion currently spawns one thread per partition and this results in poor > performance if there are more partitions than available cores/threads. It > would be better to have a thread-pool that defaults to number of available > cores. > Here is a Google doc where we can collaborate on a design discussion. > https://docs.google.com/document/d/1_wc6diy3YrRgEIhVIGzrO5AK8yhwfjWlmKtGnvbsrrY/edit?usp=sharing -- This message was sent by Atlassian Jira (v8.3.4#803005)