lxy-9602 commented on code in PR #576:
URL: https://github.com/apache/paimon-cpp/pull/576#discussion_r4614630704


##########
include/paimon/defs.h:
##########
@@ -247,6 +247,11 @@ struct PAIMON_EXPORT Options {
     /// The default value is 1024.
     static const char READ_BATCH_SIZE[];
 
+    /// "read.preparation-window" - Maximum number of data-file reader stacks 
retained per
+    /// append/raw split, including in-flight builds. Accepts positive 32-bit 
integers; defaults
+    /// to 4. Independent of reader-build executor parallelism, batch prefetch 
and warmup settings.
+    static const char READ_PREPARATION_WINDOW[];

Review Comment:
   Generally, Paimon-cpp-specific configurations are set via 
`ReadContextBuilder.SetXXX`, and `options` try to match the Java behavior. 
Please describe in the comment that the configuration here is C++-specific.



##########
src/paimon/common/reader/concat_batch_reader.cpp:
##########


Review Comment:
   Please avoid the pattern of creating an object first and then initializing 
its member variables, unless doing so would require significant code changes. 
Otherwise, prepare all members in create first and then invoke the full 
constructor.



##########
src/paimon/common/reader/concat_batch_reader.h:
##########
@@ -47,14 +48,17 @@ class ConcatBatchReader : public BatchReader {
     /// Called at most once per index, possibly concurrently. A null reader 
means a skipped file.
     using ReaderFactory = 
std::function<Result<std::unique_ptr<BatchReader>>(size_t)>;
 
-    static constexpr size_t kPreparationWindow = kReaderBuildMaxParallelNum;
+    static constexpr size_t kDefaultPreparationWindow = 4;
 
     /// Builds the initial window concurrently and refills it as readers reach 
EOF. Later
     /// construction failures are terminal read errors when the failed reader 
is reached.
+    /// The positive preparation window bounds retained readers, including 
in-flight builds,
+    /// independently of the build executor's parallelism.
     static Result<std::unique_ptr<ConcatBatchReader>> CreateLazy(
         size_t reader_count, ReaderFactory factory,
         const std::shared_ptr<arrow::MemoryPool>& arrow_pool,
-        const std::shared_ptr<Executor>& build_executor = 
GetReaderBuildExecutor());
+        const std::shared_ptr<Executor>& build_executor = 
GetReaderBuildExecutor(),
+        int32_t preparation_window = kDefaultPreparationWindow);

Review Comment:
   Please don’t use default parameters in production code.



##########
src/paimon/common/reader/concat_batch_reader.h:
##########
@@ -47,14 +48,17 @@ class ConcatBatchReader : public BatchReader {
     /// Called at most once per index, possibly concurrently. A null reader 
means a skipped file.
     using ReaderFactory = 
std::function<Result<std::unique_ptr<BatchReader>>(size_t)>;
 
-    static constexpr size_t kPreparationWindow = kReaderBuildMaxParallelNum;
+    static constexpr size_t kDefaultPreparationWindow = 4;
 
     /// Builds the initial window concurrently and refills it as readers reach 
EOF. Later
     /// construction failures are terminal read errors when the failed reader 
is reached.
+    /// The positive preparation window bounds retained readers, including 
in-flight builds,
+    /// independently of the build executor's parallelism.
     static Result<std::unique_ptr<ConcatBatchReader>> CreateLazy(
         size_t reader_count, ReaderFactory factory,
         const std::shared_ptr<arrow::MemoryPool>& arrow_pool,
-        const std::shared_ptr<Executor>& build_executor = 
GetReaderBuildExecutor());
+        const std::shared_ptr<Executor>& build_executor = 
GetReaderBuildExecutor(),
+        int32_t preparation_window = kDefaultPreparationWindow);

Review Comment:
   Also, please place parameters in the appropriate positions instead of always 
appending new parameters at the end. In paimon-cpp, executor and pool are 
typically at the end; when adding new parameters, please insert them before 
those.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to