lucasfang opened a new issue, #333: URL: https://github.com/apache/paimon-cpp/issues/333
## Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon-cpp/issues) and found nothing similar. ## Motivation Opening a data file and building its reader are dominated by remote I/O waits — an open round trip, then a footer read — but the read path did both one file at a time. Two spots serialized those waits. `PrefetchFileBatchReaderImpl::Create` opened the read-ahead cache's stream first and paid its round trip before dispatching the readers' opens. `AbstractSplitRead::CreateRawFileReaders` built the readers of a split's data files sequentially, so a split holding N files paid N sequential open-plus-footer round trips before the first batch could be produced. These are I/O-bound waits rather than CPU work, so running them concurrently cuts the wall-clock latency to the first batch, most visibly on object stores with high per-request latency. ## Solution Open all of a prefetch reader's streams in one concurrent wave, then build the readers in a second concurrent wave. The build has to be a separate wave: each build task wraps its own stream in the read-ahead cache, so folding it into the open wave would let a task block a worker of the same pool on the cache's still-queued open. Build the readers of a split's data files concurrently on a dedicated, lazily created `reader_build_executor` pool of 4 threads. It is separate from the prefetch executor to avoid a nested deadlock, and a `thread_local` guard keeps any nested `CreateRawFileReaders` reached while building a reader serial so it never blocks a worker of that same pool. `CollectAll` preserves submit order, so readers keep the split's file order, and the first error fails the whole build. `SchemaManager::ReadSchema` is already safe to call concurrently because its cache is a `ConcurrentHashMap`, so schema-evolution reads need no extra locking. The reader-build parallelism is a fixed 4 with no configuration knob. ## Anything else? No change to the storage format or protocol, and no public API change under `include/paimon/`: the reader-build parallelism is internal and is not exposed as an option. The 4-thread reader-build pool is created once per process on first use, so the pool size does not vary per split. ## Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
