lxy-9602 commented on code in PR #576:
URL: https://github.com/apache/paimon-cpp/pull/576#discussion_r4614630704
##########
include/paimon/defs.h:
##########
@@ -247,6 +247,11 @@ struct PAIMON_EXPORT Options {
/// The default value is 1024.
static const char READ_BATCH_SIZE[];
+ /// "read.preparation-window" - Maximum number of data-file reader stacks
retained per
+ /// append/raw split, including in-flight builds. Accepts positive 32-bit
integers; defaults
+ /// to 4. Independent of reader-build executor parallelism, batch prefetch
and warmup settings.
+ static const char READ_PREPARATION_WINDOW[];
Review Comment:
Generally, Paimon-cpp-specific configurations are set via
`ReadContextBuilder.SetXXX`, and `options` try to match the Java behavior.
Please describe in the comment that the configuration here is C++-specific.
##########
src/paimon/common/reader/concat_batch_reader.cpp:
##########
Review Comment:
Please avoid the pattern of creating an object first and then initializing
its member variables, unless doing so would require significant code changes.
Otherwise, prepare all members in create first and then invoke the full
constructor.
##########
src/paimon/common/reader/concat_batch_reader.h:
##########
@@ -47,14 +48,17 @@ class ConcatBatchReader : public BatchReader {
/// Called at most once per index, possibly concurrently. A null reader
means a skipped file.
using ReaderFactory =
std::function<Result<std::unique_ptr<BatchReader>>(size_t)>;
- static constexpr size_t kPreparationWindow = kReaderBuildMaxParallelNum;
+ static constexpr size_t kDefaultPreparationWindow = 4;
/// Builds the initial window concurrently and refills it as readers reach
EOF. Later
/// construction failures are terminal read errors when the failed reader
is reached.
+ /// The positive preparation window bounds retained readers, including
in-flight builds,
+ /// independently of the build executor's parallelism.
static Result<std::unique_ptr<ConcatBatchReader>> CreateLazy(
size_t reader_count, ReaderFactory factory,
const std::shared_ptr<arrow::MemoryPool>& arrow_pool,
- const std::shared_ptr<Executor>& build_executor =
GetReaderBuildExecutor());
+ const std::shared_ptr<Executor>& build_executor =
GetReaderBuildExecutor(),
+ int32_t preparation_window = kDefaultPreparationWindow);
Review Comment:
Please don’t use default parameters in production code.
##########
src/paimon/common/reader/concat_batch_reader.h:
##########
@@ -47,14 +48,17 @@ class ConcatBatchReader : public BatchReader {
/// Called at most once per index, possibly concurrently. A null reader
means a skipped file.
using ReaderFactory =
std::function<Result<std::unique_ptr<BatchReader>>(size_t)>;
- static constexpr size_t kPreparationWindow = kReaderBuildMaxParallelNum;
+ static constexpr size_t kDefaultPreparationWindow = 4;
/// Builds the initial window concurrently and refills it as readers reach
EOF. Later
/// construction failures are terminal read errors when the failed reader
is reached.
+ /// The positive preparation window bounds retained readers, including
in-flight builds,
+ /// independently of the build executor's parallelism.
static Result<std::unique_ptr<ConcatBatchReader>> CreateLazy(
size_t reader_count, ReaderFactory factory,
const std::shared_ptr<arrow::MemoryPool>& arrow_pool,
- const std::shared_ptr<Executor>& build_executor =
GetReaderBuildExecutor());
+ const std::shared_ptr<Executor>& build_executor =
GetReaderBuildExecutor(),
+ int32_t preparation_window = kDefaultPreparationWindow);
Review Comment:
Also, please place parameters in the appropriate positions instead of always
appending new parameters at the end. In paimon-cpp, executor and pool are
typically at the end; when adding new parameters, please insert them before
those.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]