[
https://issues.apache.org/jira/browse/ARROW-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16963399#comment-16963399
]
Antoine Pitrou commented on ARROW-7001:
---------------------------------------
Ok, so the problem here is to have a task blocking on another, right?
Otherwise, you can schedule a task from another task, and pass e.g. a callback
to that new task. Not very pretty, but it works.
I still think it would be worthwhile discussing this on a higher level, i.e.
which execution model we have in mind. In one execution model, IO is
interspersed with computations in the same thread and therefore a way to signal
that a task is blocking on some external completion is needed. In another
execution model, you have a driver thread that does IO and pushes available
data to pure-CPU workers.
> [C++] Develop threading APIs to accommodate nested parallelism
> ---------------------------------------------------------------
>
> Key: ARROW-7001
> URL: https://issues.apache.org/jira/browse/ARROW-7001
> Project: Apache Arrow
> Issue Type: New Feature
> Components: C++
> Reporter: Wes McKinney
> Priority: Major
>
> Tasks invoked in parallel may be able to submit their own subtasks, which in
> OpenMP and TBB documentation is often called "nested parallelism".
> If a task blocks on the completion of subtasks, then outright deadlocks are
> possible -- running tasks are all blocking on their subtasks, but the thread
> pool will not schedule any further tasks.
> I suggest that such code have a way to indicate to the thread pool (if one is
> passed in) that it is blocking on the completion of other tasks so that
> further tasks can be run while the task waits for its child tasks to
> complete. One possible way to do this is to have a floating "soft limit" for
> concurrent tasks that can be incremented when tasks are waiting.
> So if we normally allow 8 concurrent tasks, then this can be temporarily
> increased for each "suspended" task. Preferably we would provide some way for
> the dependent task group to "awaken" the suspended task so that it does not
> have to do any work while waiting for the task group to finish
> Note this feature can also be used in tasks that are waiting for IO calls
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
