SteNicholas commented on code in PR #564:
URL: https://github.com/apache/paimon-cpp/pull/564#discussion_r4597801343


##########
src/paimon/common/reader/concat_batch_reader.h:
##########
@@ -36,8 +43,22 @@ class MemoryPool;
 /// is already sorted by key and sequence number, and the key intervals do not 
overlap each other.
 class ConcatBatchReader : public BatchReader {
  public:
+    /// Must own the state needed to build the indexed reader beyond its 
caller's lifetime.
+    /// Called at most once per index, possibly concurrently. A null reader 
means a skipped file.
+    using ReaderFactory = 
std::function<Result<std::unique_ptr<BatchReader>>(size_t)>;
+
+    static constexpr size_t kPreparationWindow = kReaderBuildMaxParallelNum;

Review Comment:
   Agreed, thanks. `kPreparationWindow` currently reuses 
`kReaderBuildMaxParallelNum`, but these control different limits: the window 
bounds retained reader stacks, including in-flight preparation, while the 
executor bounds concurrent builds.
   
   A separate preparation-window option with a default of 4 makes sense as a 
follow-up. It should accept positive values and be passed into `CreateLazy` 
independently of executor parallelism and batch-prefetch/warmup settings, so 
callers can tune resource usage and preparation overlap without changing the 
build pool. We can keep the current default in this PR.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to