SteNicholas opened a new issue, #562: URL: https://github.com/apache/paimon-cpp/issues/562
### Search before asking - [x] I searched the existing issues and open pull requests and found no issue tracking per-reader reconfiguration of Arrow's global CPU pool. #287 concerns eager creation of Paimon executor threads, a different resource-lifetime problem. ### Motivation [ParquetFileBatchReader::CreateArrowReaderProperties](https://github.com/apache/paimon-cpp/blob/main/src/paimon/format/parquet/parquet_file_batch_reader.cpp#L821) reads `parquet.read.executor.thread-count`. For every nonzero value, it calls `arrow::SetCpuThreadPoolCapacity(executor_thread_count)` and then enables threaded column reads. The option is supplied while building a file reader, but the setter changes a process-wide Arrow CPU pool. As a result: 1. Constructing a reader with the default value can override a pool capacity configured by the embedding application. 2. Building readers with different nonzero values repeatedly changes the scheduler shared by readers that are already running. 3. [Parallel prefetch reader construction](https://github.com/apache/paimon-cpp/blob/main/src/paimon/common/reader/prefetch_file_batch_reader_impl.cpp#L284) repeats this global operation for each sub-reader, while outer Paimon parallelism and Arrow column parallelism are configured separately. 4. [WriteContext spill setup](https://github.com/apache/paimon-cpp/blob/main/src/paimon/core/operation/write_context.cpp#L292) also configures the same global CPU pool, so read-side construction can interfere with that setting. A minimal observable scenario is to configure the Arrow CPU pool in the host application, build a Parquet reader with a different nonzero thread count, and inspect `arrow::GetCpuThreadPoolCapacity()`. The capacity changes during reader setup, affecting the process rather than just the new reader. ### Solution Separate reader parallelism from ownership of the process-wide scheduler: - Remove routine mutation of the global Arrow CPU pool from file-reader construction. - Define one documented owner and scope for CPU capacity configuration. Audit the executor facilities available in the supported Arrow version before selecting the implementation. - If scoped decoding executors can be supported, use an appropriate read/context-level budget together with existing outer prefetch parallelism. If capacity remains process-wide, make that contract explicit and manage it through a deliberate application/global configuration path. - Preserve the zero value's role in disabling threaded reads. Explicitly document and test the migration of nonzero values instead of silently ignoring them or making the first reader's value win through call_once. - Keep the change focused on scheduler configuration; no decoding or storage-format change is needed. The existing Parquet reader tests assert changes to global capacity, so this is a concurrency/configuration enhancement requiring an explicit compatibility decision rather than an undocumented behavior change. ### Anything else? Acceptance and validation: - [ ] Creating, reading, and destroying a Parquet reader does not unexpectedly resize an application-configured global CPU pool. - [ ] Readers/contexts with different settings can coexist with deterministic, documented behavior. - [ ] Cover zero/default/nonzero settings and parallel creation of prefetch sub-readers; do not introduce nested-executor deadlocks. - [ ] Cover interaction with other Arrow CPU-pool consumers, including configured spill operations. - [ ] Update tests and option documentation to match the chosen scope and compatibility policy. - [ ] Compare multi-reader throughput, CPU utilization, and latency under concurrent workloads before selecting defaults. The setter is a global resource side effect, not evidence of a data race or data corruption. The proposal is based on source inspection; no benchmark result is claimed. ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
