[
https://issues.apache.org/jira/browse/ARROW-11588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weston Pace updated ARROW-11588:
--------------------------------
Description:
There are a few places we queue data/jobs today (e.g. readahead, thread pool).
In order to reduce complexity (single responsibility) and add traceability (so
we can tell where queues are filling up) we should create general purpose
queues (e.g. the boost lockfree one) and add some basic performance
counters/events (queue size, queue full, queue starved).
We will need multiple queue implementations. Some places will need a spsc lock
free queue and others will need a multi-producer / multi-consumer queue with
proper waits (and potentially more queues than that).
was:There are a few places we queue data/jobs today (e.g. readahead, thread
pool). In order to reduce complexity (adopt a single implementation where we
have multiple) and add traceability (so we can tell where queues are filling
up) we should adopt a general purpose queue (e.g. the boost lockfree one) and
add some basic performance counters/events (queue size, queue full, queue
starved).
> [C++] Add traceable general purpose queue(s)
> --------------------------------------------
>
> Key: ARROW-11588
> URL: https://issues.apache.org/jira/browse/ARROW-11588
> Project: Apache Arrow
> Issue Type: Task
> Components: C++
> Reporter: Weston Pace
> Assignee: Weston Pace
> Priority: Major
>
> There are a few places we queue data/jobs today (e.g. readahead, thread
> pool). In order to reduce complexity (single responsibility) and add
> traceability (so we can tell where queues are filling up) we should create
> general purpose queues (e.g. the boost lockfree one) and add some basic
> performance counters/events (queue size, queue full, queue starved).
> We will need multiple queue implementations. Some places will need a spsc
> lock free queue and others will need a multi-producer / multi-consumer queue
> with proper waits (and potentially more queues than that).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
