SteNicholas commented on code in PR #564:
URL: https://github.com/apache/paimon-cpp/pull/564#discussion_r4597801343
##########
src/paimon/common/reader/concat_batch_reader.h:
##########
@@ -36,8 +43,22 @@ class MemoryPool;
/// is already sorted by key and sequence number, and the key intervals do not
overlap each other.
class ConcatBatchReader : public BatchReader {
public:
+ /// Must own the state needed to build the indexed reader beyond its
caller's lifetime.
+ /// Called at most once per index, possibly concurrently. A null reader
means a skipped file.
+ using ReaderFactory =
std::function<Result<std::unique_ptr<BatchReader>>(size_t)>;
+
+ static constexpr size_t kPreparationWindow = kReaderBuildMaxParallelNum;
Review Comment:
Agreed, thanks. `kPreparationWindow` currently reuses
`kReaderBuildMaxParallelNum`, but these control different limits: the window
bounds retained reader stacks, including in-flight preparation, while the
executor bounds concurrent builds.
A separate preparation-window option with a default of 4 makes sense as a
follow-up. It should accept positive values and be passed into `CreateLazy`
independently of executor parallelism and batch-prefetch/warmup settings, so
callers can tune resource usage and preparation overlap without changing the
build pool. We can keep the current default in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]