Weston Pace created ARROW-12097:
-----------------------------------
Summary: [C++] Modify BackgroundGenerator so it creates fewer
threads
Key: ARROW-12097
URL: https://issues.apache.org/jira/browse/ARROW-12097
Project: Apache Arrow
Issue Type: Improvement
Components: C++
Reporter: Weston Pace
Assignee: Weston Pace
Fix For: 4.0.0
The current implementation creates a thread per block and in the CSV reader
this hurts performance just a bit. However, in the IPC reader this hurts
performance even more.
Instead the readahead can move inside the background generator and the
background generator task can keep running until the queue fills up and then
restart when the queue has drained enough for a substantial amount of work to
be done.
In my test CSV case this dropped the # of thread tasks created from ~2.5k to
~100.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
