[
https://issues.apache.org/jira/browse/ARROW-10038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17198490#comment-17198490
]
Adam Hooper commented on ARROW-10038:
-------------------------------------
> While it may "go against the spirit", is there any concrete problem with it?
Not that I can find. I filed this bug before I realized that I can write code
that avoids using the global thread pool entirely.
My main beef is confidence -- as an application author, I lose confidence
{{libarrow.a}} will behave as I instruct it.
My use case, by the way: I create small executables that manipulate
Arrow/Parquet files and then exit. Most files are small (0-100MB) and already
in RAM (because another program just wrote them). So I expect -- though I
haven't benchmarked -- that if I start my programs 100,000 times, pthreads
would take a noticeable chunk of time.
But again: I can avoid the global thread pool, so this doesn't affect me.
> > API users would be expected to set the thread-pool capacity before creating
> > the thread pool.
> That's undesirable. The global thread pool can be created implicitly, and
> users must be able to change the thread count afterwards.
As a library user, this isn't a feature I would expect or feel I deserve.
Go uses {{GOMAXPROCS}} (can't be resized); Openblas uses
{{OPENBLAS_NUM_THREADS}} (can't be resized); OpenMP uses {{OMP_NUM_THREADS}};
Python/Java have no global thread pool (and Python ThreadPoolExecutor can't be
resized); Haskell uses {{+RTS Ncores}} cmdline argument.... I spent half an
hour searching just to find an example that _does_ let users change the a
global thread pool's thread count: C# CLR.
(OpenMP actually uses my second idea -- {{omp_set_num_threads()}} calls
{{setenv()}}.)
> SetCpuThreadPoolCapacity(1) spins up nCPUs threads
> --------------------------------------------------
>
> Key: ARROW-10038
> URL: https://issues.apache.org/jira/browse/ARROW-10038
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++
> Affects Versions: 1.0.1
> Reporter: Adam Hooper
> Priority: Minor
>
> When I call {{arrow::SetCpuThreadPoolCapacity(1);}}, Arrow does this:
> 1. Spins up a singleton {{ThreadPool}} with _the default thread count_;
> 2. Sets the number of threads on that {{ThreadPool}} to 1 -- killing the
> extra threads.
> On my Intel system, I'm forced to spin up four threads to set the CPU
> thread-pool capacity to 1. This goes against the spirit of the API method --
> or at least, my understanding of it (and my experience with other thread
> pools).
> My workaround, for calling code: instead of calling
> {{arrow::SetCpuThreadPoolCapacity(1)}}, call {{setenv("OMP_NUM_THREADS", "1",
> 1)}}.
> Brainstorming, here are some ideas for Arrow's global thread pool that would
> stop launching {{>limit}} threads to set the limit:
> * {{cpu_thread_pool_capacity}} could be a global variable, not an attribute
> on the global {{ThreadPool}}. API users would be expected to set the
> thread-pool capacity _before_ creating the thread pool. (They're probably
> doing this anyway.)
> * {{SetCpuThreadPoolCapacity()}} could call {{setenv("OMP_NUM_THREADS", ...)}}
> * {{ThreadPool}} could create threads on-demand instead of in the ctor. An
> unused {{ThreadPool}} would launch zero threads -- resolving ARROW-10033 as a
> side-effect
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
