[
https://issues.apache.org/jira/browse/ARROW-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17291966#comment-17291966
]
Weston Pace commented on ARROW-7001:
------------------------------------
Thanks [~anton-malakhov] . I took a look at the blog and oox looks like it can
be useful. I have been working on this issue and a slightly broader goal of
removing all blocking I/O waits from the CPU thread pool (see
[https://docs.google.com/document/d/1tO2WwYL-G2cB_MCPqYguKjKkRT7mZ8C2Gc9ONvspfgo/edit?usp=sharing]
).
I have been using an arrow::Result-aware implementation of std::future
(arrow::Future) and using simple Then/All/Any continuations. This only allows
for dags of flow dependencies (and not output & anti dependencies). I'm fairly
certain this will be sufficient and there is a broad understanding of the
capabilities from other languages that support this kind of construct (e.g.
Javascript's Promise.then, C# Task.ContinueWith, Java's CompletableFuture).
I was not able to find any oox library or mention of licensing. If there is a
repository out there somewhere with a reference implementation I'd be happy to
take a closer look. I appreciate you bringing sharing this but at the moment I
do not think the sophistication offered by oox is needed to solve this problem.
> [C++] Develop threading APIs to accommodate nested parallelism
> ---------------------------------------------------------------
>
> Key: ARROW-7001
> URL: https://issues.apache.org/jira/browse/ARROW-7001
> Project: Apache Arrow
> Issue Type: New Feature
> Components: C++
> Reporter: Wes McKinney
> Assignee: Weston Pace
> Priority: Major
> Fix For: 4.0.0
>
>
> Tasks invoked in parallel may be able to submit their own subtasks, which in
> OpenMP and TBB documentation is often called "nested parallelism".
> If a task blocks on the completion of subtasks, then outright deadlocks are
> possible -- running tasks are all blocking on their subtasks, but the thread
> pool will not schedule any further tasks.
> I suggest that such code have a way to indicate to the thread pool (if one is
> passed in) that it is blocking on the completion of other tasks so that
> further tasks can be run while the task waits for its child tasks to
> complete. One possible way to do this is to have a floating "soft limit" for
> concurrent tasks that can be incremented when tasks are waiting.
> So if we normally allow 8 concurrent tasks, then this can be temporarily
> increased for each "suspended" task. Preferably we would provide some way for
> the dependent task group to "awaken" the suspended task so that it does not
> have to do any work while waiting for the task group to finish
> Note this feature can also be used in tasks that are waiting for IO calls
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
