[
https://issues.apache.org/jira/browse/ARROW-8667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Farmer reassigned ARROW-8667:
----------------------------------
Assignee: (was: Weston Pace)
This issue was last updated over 90 days ago, which may be an indication it is
no longer being actively worked. To better reflect the current state, the issue
is being unassigned. Please feel free to re-take assignment of the issue if it
is being actively worked, or if you plan to start that work soon.
> [C++] Add multi-consumer Scheduler API to sit one layer above ThreadPool
> ------------------------------------------------------------------------
>
> Key: ARROW-8667
> URL: https://issues.apache.org/jira/browse/ARROW-8667
> Project: Apache Arrow
> Issue Type: New Feature
> Components: C++
> Reporter: Wes McKinney
> Priority: Major
>
> I believe we should define an abstraction to allow for custom resource
> allocation strategies (round robin, even time, etc.) to be devised for
> situations where there are different thread pool consumers that are working
> independently of each other.
> Consider the classic nested parallelism scenario:
> * Task A in thread 1 may issue N subtasks that run in parallel
> * Task B in thread 2 may issue K subtasks
> With our current ThreadPool abstraction, it is easy to conceive scenarios
> where either Task A or Task B trample each other.
> One approach to remedy this problem is to have an API like so:
> {code}
> // Inform the scheduler that you want to submit tasks that are "your tasks"
> Consumer* consumer = scheduler->NewConsumer();
> for (...) {
> Future<T> fut = scheduler->Submit(consumer, DoWork, ...);
> }
> // Join for all tasks to finish.
> consumer->Finish();
> {code}
> The idea is that the scheduler would maintain separate task queues for each
> consumer and e.g. track consumer-specific metrics of interest to determine
> how tasks are allocated.
> The scheduler could have different logic to control tasks being assigned to
> worker threads:
> * Round-robin
> * Even-time allocation (run fewer tasks for consumers with "slow" tasks and
> more tasks from consumers with "fast" tasks -- though there are some nuances
> here like avoiding starving a consumer if they've been doing a lot of "slow"
> tasks and then a "fast" consumer shows up)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
