[
https://issues.apache.org/jira/browse/ARROW-12255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andy Grove closed ARROW-12255.
------------------------------
Resolution: Won't Fix
Moved to https://github.com/apache/arrow-datafusion/issues/64
> [Rust] [Ballista] Integrate scheduler with DataFusion
> -----------------------------------------------------
>
> Key: ARROW-12255
> URL: https://issues.apache.org/jira/browse/ARROW-12255
> Project: Apache Arrow
> Issue Type: New Feature
> Components: Rust - Ballista, Rust - DataFusion
> Reporter: Andy Grove
> Assignee: Andy Grove
> Priority: Major
> Fix For: 5.0.0
>
>
> The Ballista scheduler breaks a query down into stages based on changes in
> partitioning int he plan, where each stage is broken down into tasks that can
> be executed concurrently.
> Rather than trying to run all the partitions at once, Ballista executors
> process n concurrent tasks at a time and then request new tasks from the
> scheduler.
> This approach would help DataFusion scale better and it would be ideal to use
> the same scheduler to scale across cores in DataFusion and across nodes in
> Ballista.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
