[jira] [Commented] (ARROW-8767) [C++] Make ThreadPool task ordering configurable

2020-05-12 Thread David Li (Jira) 


[ 
https://issues.apache.org/jira/browse/ARROW-8767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105391#comment-17105391
 ] 

David Li commented on ARROW-8767:
-

For I/O, it would also be useful to be able to assign a "weight" to a task 
(representing bandwidth or some other non-CPU resource) so a scheduler could 
spawn tasks based on that metric. (For instance, you may wish to allow more 
concurrent tasks when weights are small, or throttle a consumer that is 
consuming disproportionate resources.)

> [C++] Make ThreadPool task ordering configurable
> 
>
> Key: ARROW-8767
> URL: https://issues.apache.org/jira/browse/ARROW-8767
> Project: Apache Arrow
>  Issue Type: Wish
>  Components: C++
>Reporter: Antoine Pitrou
>Priority: Major
>
> We may want to choose a task ordering strategy when constructing a ThreadPool.
> To make the ordering strategy configurable, we may want to externalize it in 
> a separate JobQueue class.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (ARROW-8767) [C++] Make ThreadPool task ordering configurable

2020-05-11 Thread Wes McKinney (Jira) 


[ 
https://issues.apache.org/jira/browse/ARROW-8767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17104623#comment-17104623
 ] 

Wes McKinney commented on ARROW-8767:
-

I linked a couple of related issues. Whatever solution is devised should 
hopefully be able to accommodate the round-robin strategy and the 
fair-time-allocation strategy (wherein consumers' total CPU time is tracked and 
used to allocate CPU capacity to consumers which have not had their fair share)

> [C++] Make ThreadPool task ordering configurable
> 
>
> Key: ARROW-8767
> URL: https://issues.apache.org/jira/browse/ARROW-8767
> Project: Apache Arrow
>  Issue Type: Wish
>  Components: C++
>Reporter: Antoine Pitrou
>Priority: Major
>
> We may want to choose a task ordering strategy when constructing a ThreadPool.
> To make the ordering strategy configurable, we may want to externalize it in 
> a separate JobQueue class.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (ARROW-8767) [C++] Make ThreadPool task ordering configurable

2020-05-11 Thread Antoine Pitrou (Jira) 


[ 
https://issues.apache.org/jira/browse/ARROW-8767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17104610#comment-17104610
 ] 

Antoine Pitrou commented on ARROW-8767:
---

cc [~fsaintjacques]

Possible API:

{code:cpp}
struct TaskSpec {
  int64_t priority;
};

class ThreadPool {
 public:
  Status Spawn(TaskSpec, ...);
};
{code}


> [C++] Make ThreadPool task ordering configurable
> 
>
> Key: ARROW-8767
> URL: https://issues.apache.org/jira/browse/ARROW-8767
> Project: Apache Arrow
>  Issue Type: Wish
>  Components: C++
>Reporter: Antoine Pitrou
>Priority: Major
>
> We may want to choose a task ordering strategy when constructing a ThreadPool.
> To make the ordering strategy configurable, we may want to externalize it in 
> a separate JobQueue class.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)