lucasfang commented on code in PR #564:
URL: https://github.com/apache/paimon-cpp/pull/564#discussion_r4597619301


##########
src/paimon/common/reader/concat_batch_reader.h:
##########
@@ -36,8 +43,22 @@ class MemoryPool;
 /// is already sorted by key and sequence number, and the key intervals do not 
overlap each other.
 class ConcatBatchReader : public BatchReader {
  public:
+    /// Must own the state needed to build the indexed reader beyond its 
caller's lifetime.
+    /// Called at most once per index, possibly concurrently. A null reader 
means a skipped file.
+    using ReaderFactory = 
std::function<Result<std::unique_ptr<BatchReader>>(size_t)>;
+
+    static constexpr size_t kPreparationWindow = kReaderBuildMaxParallelNum;

Review Comment:
   Non-blocking suggestion: could we make the reader preparation window 
configurable, keeping 4 as the default? Different file sizes and storage 
latencies may benefit from different trade-offs between retained resources and 
preparation overlap. It would also be useful to tune this independently of the 
reader-build executor's parallelism. This could be a follow-up.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to