SteNicholas opened a new pull request, #574:
URL: https://github.com/apache/paimon-cpp/pull/574

   <!-- PR titles must follow Conventional Commits: <type>(<optional-scope>): 
<description> -->
   
   ### Purpose
   
   <!-- Linking this pull request to the issue -->
   Linked issue: close #562
   
   <!-- What is the purpose of the change -->
   Parquet readers resized Arrow's global CPU pool during construction, 
including when using the
   default options. This overwrote capacity configured by the embedding 
application or spill
   setup and made readers with different settings depend on construction order. 
Parallel
   prefetch construction repeated the same process-wide change for each 
sub-reader.
   
   The change separates reader column parallelism from ownership of the shared 
CPU pool:
   
   - **`ParquetFileBatchReader`**: removes global CPU-pool resizing from reader 
construction and
     adds `parquet.read.use-threads` to control parallel column decoding. 
Threaded readers use
     the existing Arrow CPU pool; creating, reading and closing readers 
preserve its capacity.
   - **Legacy option compatibility**: deprecates 
`parquet.read.executor.thread-count` as a
     zero/nonzero fallback when the boolean option is absent. Zero disables 
column threading;
     positive values enable it without setting global capacity. The boolean 
option takes
     precedence, and both supplied options are validated, including an 
overridden legacy value.
   - **CPU capacity ownership**: leaves capacity configuration to the 
application. The existing
     positive spill-thread setting remains an explicit process-wide setter 
applied by
     `WriteContextBuilder::Finish()`; subsequent Parquet readers preserve that 
capacity.
     Paimon's outer prefetch executor remains independent of Arrow's 
column-decoding pool.
   - **Concurrent-reader benchmarks**: compare one and four readers using 
default, zero,
     nonzero and mixed legacy settings. Counters cover aggregate throughput, 
mean reader
     open/scan/close latency and effective CPU capacity, alongside process CPU 
time.
   
   Arrow 17's synchronous batch-reader API does not accept a scoped CPU 
executor. The change
   retains that read path and its decoding behavior. The default still enables 
column threading,
   but no longer forces a global capacity of three; applications needing a 
fixed CPU budget
   configure the shared pool before starting concurrent work.
   
   ### Tests
   
   <!-- Describe tests added or updated and what they cover. -->
   UT:
   
   - `src/paimon/format/parquet/parquet_file_batch_reader_test.cpp`:
     - `TestCreateArrowReaderProperties`: consolidates default, zero, positive 
and explicit
       boolean settings into table-driven cases. It checks option precedence, 
unsigned 32-bit
       boundaries, invalid values and unchanged application CPU capacity while 
retaining the
       existing batch-size, pre-buffer and cache-option assertions.
     - `TestConcurrentReadersPreserveGlobalCpuPool`: constructs readers with 
different settings
       concurrently, reads them alongside Arrow IPC spill write/read 
operations, and checks
       returned data and capacity through reader close and destruction. It 
covers both disabled
       and enabled spill threading and restores the original global capacity.
     - `TestPrefetchReadersPreserveGlobalCpuPool`: exercises prefetch 
sub-reader construction
       and decoded warmup with one and four outer workers, read-ahead enabled 
and disabled, and
       default, legacy and boolean options. It checks complete output and 
capacity preservation
       with a single-worker Arrow CPU pool, including after reader destruction.
   
   IT:
   
   No standalone integration tests are added. The reader tests exercise real 
Parquet files
   through `ParquetReaderBuilder` and `PrefetchFileBatchReaderImpl`, and real 
Arrow IPC spill
   files using the threading policy from `WriteContextBuilder`.
   
   ### API and Format
   
   <!-- Does this change affect API in include dir or storage format or 
protocol -->
   No public API signatures, stored file formats or table protocols change.
   `include/paimon/write_context.h` clarifies the existing spill setter's 
process-wide scope.
   
   The new `parquet.read.use-threads` option defaults to enabled. The 
deprecated numeric option
   continues to control whether column threading is enabled when the boolean 
option is absent,
   but positive values no longer configure a thread count. Applications that 
relied on those
   values to size the global pool must move capacity configuration to 
application startup.
   
   ### Documentation
   
   <!-- Does this change introduce a new feature -->
   `docs/source/user_guide/read.rst` documents application ownership of Arrow 
CPU capacity,
   legacy-option migration, spill interaction, the independent prefetch 
executor and the
   constraint against submitting synchronous threaded reads to the same 
saturated Arrow pool.
   `docs/source/examples/benchmark.rst` documents the concurrent-reader 
policies, counters
   and before/after methodology. It calls for separate processes per comparison 
case and
   distinguishes a change in effective CPU budget from a fixed-budget speedup.
   
   ### Generative AI tooling
   
   Generated-by: Codex (GPT-6)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to