lucasfang opened a new pull request, #334: URL: https://github.com/apache/paimon-cpp/pull/334
### Purpose Linked issue: close #333 Opening a data file and building its reader are dominated by remote I/O waits — an open round trip, then a footer read — but the read path did both one file at a time. `PrefetchFileBatchReaderImpl::Create` opened the read-ahead cache's stream first and paid its round trip before dispatching the readers' opens, and `AbstractSplitRead::CreateRawFileReaders` built the readers of a split's data files sequentially, so a split holding N files paid N sequential open-plus-footer round trips before the first batch. Since these are I/O-bound waits rather than CPU work, this PR runs them concurrently to cut the latency to the first batch, most visibly on object stores with high per-request latency. A prefetch reader now opens all of its streams in one concurrent wave, then builds the readers in a second concurrent wave. The build has to be a separate wave: each build task wraps its own stream in the read-ahead cache, so folding it into the open wave would let a task block a worker of the same pool on the cache's still-queued open. Both waves run on the read context's executor, which is sized to the prefetch parallelism, so no pool is spun up and torn down per file and the open/build concurrency follows the configured parallelism instead of a fixed default; that reuse is safe because neither wave's tasks wait on one another or on that executor, and `Create` never runs on one of its workers. The readers of a split's data files are built concurrently on a dedicated, lazily created `reader_build_executor` pool of 4 threads, separate from the read context executor the inner waves use so the two never block on the same pool; a `thread_local` guard keeps any nested `CreateRawFileRe aders` reached while building a reader serial so it never blocks a worker of that same pool. `CollectAll` preserves submit order, so readers keep the split's file order and the first error fails the whole build. `SchemaManager::ReadSchema` is already safe to call concurrently because its cache is a `ConcurrentHashMap`, so schema-evolution reads need no extra locking. The reader-build parallelism is a fixed 4 with no configuration knob. The 4-thread pool is created once per process on first use, so its size does not vary per split; a split with fewer files builds only that many readers. ### Tests New `PrefetchFileBatchReaderImplTest.TestCreateOpensAllStreamsConcurrently` uses a barrier file system that fails unless every stream (one for the cache plus one per reader) is in flight at once, pinning down that the cache's open is no longer serialized ahead of the readers'. `ScanAndReadInteTest.TestWithAppendSnapshot5WithReaderBuildParallelism` reads a snapshot-5 bucket holding five files through the concurrent build, and `ReadInteTest.TestAppendReadWithSchemaEvolutionWithReaderBuildParallelism` reads splits that mix two schema ids per split, exercising concurrent `SchemaManager::ReadSchema`. Verified in a Debug build (`-j 96`): - `paimon-core-test --gtest_filter='CoreOptionsTest.*'`: 39 tests passed. - `paimon-scan-and-read-inte-test --gtest_filter='*TestWithAppendSnapshot5WithReaderBuildParallelism*:*TestWithAppendSnapshot5/*'`: 16 tests passed. - `paimon-read-inte-test --gtest_filter='*SchemaEvolution*'`: 97 tests passed. - `pre-commit run --files <changed files>` and `git diff --check`: clean. ### API and Format No. `include/paimon/defs.h` is byte-identical to main — the previously drafted `read.reader-build.max-parallel-num` option and `CoreOptions::GetReaderBuildMaxParallelNum()` were dropped, and the parallelism is now an internal fixed 4 rather than a public option. No storage format or protocol change. ### Documentation No. This is an internal read-path optimization with no user-visible API, configuration, or behavior change. ### Generative AI tooling Generated-by: Qoder -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
