[
https://issues.apache.org/jira/browse/ARROW-8774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17105527#comment-17105527
]
Wes McKinney edited comment on ARROW-8774 at 5/12/20, 3:29 PM:
---------------------------------------------------------------
FYI we're also working on our thread scheduling and concurrent CPU/IO
management APIs for C++ for nearly the exact same reasons you are. We might try
to see if there are shared learnings, we can at least link issues on JIRA to
point our design docs or work happening
cc [~fsaintjacques] [~lidavidm] [~apitrou]
was (Author: wesmckinn):
FYI we're also working on our thread scheduling and concurrent CPU/IO
management APIs for C++. We might try to see if there are shared learnings, we
can at least link issues on JIRA to point our design docs or work happening
cc [~fsaintjacques] [~lidavidm] [~apitrou]
> [Rust] [DataFusion] Improve threading model
> -------------------------------------------
>
> Key: ARROW-8774
> URL: https://issues.apache.org/jira/browse/ARROW-8774
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Rust, Rust - DataFusion
> Reporter: Andy Grove
> Assignee: Andy Grove
> Priority: Major
> Fix For: 1.0.0
>
>
> DataFusion currently spawns one thread per partition and this results in poor
> performance if there are more partitions than available cores/threads. It
> would be better to have a thread-pool that defaults to number of available
> cores.
> Here is a Google doc where we can collaborate on a design discussion.
> https://docs.google.com/document/d/1_wc6diy3YrRgEIhVIGzrO5AK8yhwfjWlmKtGnvbsrrY/edit?usp=sharing
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
