Weston Pace created ARROW-15035:
-----------------------------------
Summary: [C++] Create I/O thread pools per filesystem
Key: ARROW-15035
URL: https://issues.apache.org/jira/browse/ARROW-15035
Project: Apache Arrow
Issue Type: Improvement
Components: C++
Reporter: Weston Pace
The IOContext gets us most of the way here but we still don't do this yet
today. One concrete advantage to this is it allows us to more intelligently
set the number of I/O threads.
For example, 8 threads is often too small for an S3 filesystem (ARROW-14965)
On the other hand, in some cases, 8 threads can be too many for an HDD
(ARROW-14354)
I doubt we will be able to figure out the ideal size of the I/O thread pool for
any filesystem (e.g. on an S3 filesystem it depends on how many cores you have
and how much bandwidth the system has) but we can possibly have more sensible
defaults.
Furthermore, it will hopefully clarify to the user the connection between
filesystem and I/O thread pool size.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
