SteNicholas opened a new issue, #562:
URL: https://github.com/apache/paimon-cpp/issues/562

   ### Search before asking
   
   - [x] I searched the existing issues and open pull requests and found no 
issue tracking per-reader reconfiguration of Arrow's global CPU pool. #287 
concerns eager creation of Paimon executor threads, a different 
resource-lifetime problem.
   
   ### Motivation
   
   
[ParquetFileBatchReader::CreateArrowReaderProperties](https://github.com/apache/paimon-cpp/blob/main/src/paimon/format/parquet/parquet_file_batch_reader.cpp#L821)
 reads `parquet.read.executor.thread-count`. For every nonzero value, it calls 
`arrow::SetCpuThreadPoolCapacity(executor_thread_count)` and then enables 
threaded column reads.
   
   The option is supplied while building a file reader, but the setter changes 
a process-wide Arrow CPU pool. As a result:
   
   1. Constructing a reader with the default value can override a pool capacity 
configured by the embedding application.
   2. Building readers with different nonzero values repeatedly changes the 
scheduler shared by readers that are already running.
   3. [Parallel prefetch reader 
construction](https://github.com/apache/paimon-cpp/blob/main/src/paimon/common/reader/prefetch_file_batch_reader_impl.cpp#L284)
 repeats this global operation for each sub-reader, while outer Paimon 
parallelism and Arrow column parallelism are configured separately.
   4. [WriteContext spill 
setup](https://github.com/apache/paimon-cpp/blob/main/src/paimon/core/operation/write_context.cpp#L292)
 also configures the same global CPU pool, so read-side construction can 
interfere with that setting.
   
   A minimal observable scenario is to configure the Arrow CPU pool in the host 
application, build a Parquet reader with a different nonzero thread count, and 
inspect `arrow::GetCpuThreadPoolCapacity()`. The capacity changes during reader 
setup, affecting the process rather than just the new reader.
   
   ### Solution
   
   Separate reader parallelism from ownership of the process-wide scheduler:
   
   - Remove routine mutation of the global Arrow CPU pool from file-reader 
construction.
   - Define one documented owner and scope for CPU capacity configuration. 
Audit the executor facilities available in the supported Arrow version before 
selecting the implementation.
   - If scoped decoding executors can be supported, use an appropriate 
read/context-level budget together with existing outer prefetch parallelism. If 
capacity remains process-wide, make that contract explicit and manage it 
through a deliberate application/global configuration path.
   - Preserve the zero value's role in disabling threaded reads. Explicitly 
document and test the migration of nonzero values instead of silently ignoring 
them or making the first reader's value win through call_once.
   - Keep the change focused on scheduler configuration; no decoding or 
storage-format change is needed.
   
   The existing Parquet reader tests assert changes to global capacity, so this 
is a concurrency/configuration enhancement requiring an explicit compatibility 
decision rather than an undocumented behavior change.
   
   ### Anything else?
   
   Acceptance and validation:
   
   - [ ] Creating, reading, and destroying a Parquet reader does not 
unexpectedly resize an application-configured global CPU pool.
   - [ ] Readers/contexts with different settings can coexist with 
deterministic, documented behavior.
   - [ ] Cover zero/default/nonzero settings and parallel creation of prefetch 
sub-readers; do not introduce nested-executor deadlocks.
   - [ ] Cover interaction with other Arrow CPU-pool consumers, including 
configured spill operations.
   - [ ] Update tests and option documentation to match the chosen scope and 
compatibility policy.
   - [ ] Compare multi-reader throughput, CPU utilization, and latency under 
concurrent workloads before selecting defaults.
   
   The setter is a global resource side effect, not evidence of a data race or 
data corruption. The proposal is based on source inspection; no benchmark 
result is claimed.
   
   ### Are you willing to submit a PR?
   
   - [ ] I'm willing to submit a PR!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to