SteNicholas opened a new pull request, #574:
URL: https://github.com/apache/paimon-cpp/pull/574
<!-- PR titles must follow Conventional Commits: <type>(<optional-scope>):
<description> -->
### Purpose
<!-- Linking this pull request to the issue -->
Linked issue: close #562
<!-- What is the purpose of the change -->
Parquet readers resized Arrow's global CPU pool during construction,
including when using the
default options. This overwrote capacity configured by the embedding
application or spill
setup and made readers with different settings depend on construction order.
Parallel
prefetch construction repeated the same process-wide change for each
sub-reader.
The change separates reader column parallelism from ownership of the shared
CPU pool:
- **`ParquetFileBatchReader`**: removes global CPU-pool resizing from reader
construction and
adds `parquet.read.use-threads` to control parallel column decoding.
Threaded readers use
the existing Arrow CPU pool; creating, reading and closing readers
preserve its capacity.
- **Legacy option compatibility**: deprecates
`parquet.read.executor.thread-count` as a
zero/nonzero fallback when the boolean option is absent. Zero disables
column threading;
positive values enable it without setting global capacity. The boolean
option takes
precedence, and both supplied options are validated, including an
overridden legacy value.
- **CPU capacity ownership**: leaves capacity configuration to the
application. The existing
positive spill-thread setting remains an explicit process-wide setter
applied by
`WriteContextBuilder::Finish()`; subsequent Parquet readers preserve that
capacity.
Paimon's outer prefetch executor remains independent of Arrow's
column-decoding pool.
- **Concurrent-reader benchmarks**: compare one and four readers using
default, zero,
nonzero and mixed legacy settings. Counters cover aggregate throughput,
mean reader
open/scan/close latency and effective CPU capacity, alongside process CPU
time.
Arrow 17's synchronous batch-reader API does not accept a scoped CPU
executor. The change
retains that read path and its decoding behavior. The default still enables
column threading,
but no longer forces a global capacity of three; applications needing a
fixed CPU budget
configure the shared pool before starting concurrent work.
### Tests
<!-- Describe tests added or updated and what they cover. -->
UT:
- `src/paimon/format/parquet/parquet_file_batch_reader_test.cpp`:
- `TestCreateArrowReaderProperties`: consolidates default, zero, positive
and explicit
boolean settings into table-driven cases. It checks option precedence,
unsigned 32-bit
boundaries, invalid values and unchanged application CPU capacity while
retaining the
existing batch-size, pre-buffer and cache-option assertions.
- `TestConcurrentReadersPreserveGlobalCpuPool`: constructs readers with
different settings
concurrently, reads them alongside Arrow IPC spill write/read
operations, and checks
returned data and capacity through reader close and destruction. It
covers both disabled
and enabled spill threading and restores the original global capacity.
- `TestPrefetchReadersPreserveGlobalCpuPool`: exercises prefetch
sub-reader construction
and decoded warmup with one and four outer workers, read-ahead enabled
and disabled, and
default, legacy and boolean options. It checks complete output and
capacity preservation
with a single-worker Arrow CPU pool, including after reader destruction.
IT:
No standalone integration tests are added. The reader tests exercise real
Parquet files
through `ParquetReaderBuilder` and `PrefetchFileBatchReaderImpl`, and real
Arrow IPC spill
files using the threading policy from `WriteContextBuilder`.
### API and Format
<!-- Does this change affect API in include dir or storage format or
protocol -->
No public API signatures, stored file formats or table protocols change.
`include/paimon/write_context.h` clarifies the existing spill setter's
process-wide scope.
The new `parquet.read.use-threads` option defaults to enabled. The
deprecated numeric option
continues to control whether column threading is enabled when the boolean
option is absent,
but positive values no longer configure a thread count. Applications that
relied on those
values to size the global pool must move capacity configuration to
application startup.
### Documentation
<!-- Does this change introduce a new feature -->
`docs/source/user_guide/read.rst` documents application ownership of Arrow
CPU capacity,
legacy-option migration, spill interaction, the independent prefetch
executor and the
constraint against submitting synchronous threaded reads to the same
saturated Arrow pool.
`docs/source/examples/benchmark.rst` documents the concurrent-reader
policies, counters
and before/after methodology. It calls for separate processes per comparison
case and
distinguishes a change in effective CPU budget from a fixed-budget speedup.
### Generative AI tooling
Generated-by: Codex (GPT-6)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]