SteNicholas commented on code in PR #576:
URL: https://github.com/apache/paimon-cpp/pull/576#discussion_r4620845131
##########
src/paimon/common/reader/concat_batch_reader.h:
##########
@@ -47,14 +48,17 @@ class ConcatBatchReader : public BatchReader {
/// Called at most once per index, possibly concurrently. A null reader
means a skipped file.
using ReaderFactory =
std::function<Result<std::unique_ptr<BatchReader>>(size_t)>;
- static constexpr size_t kPreparationWindow = kReaderBuildMaxParallelNum;
+ static constexpr size_t kDefaultPreparationWindow = 4;
/// Builds the initial window concurrently and refills it as readers reach
EOF. Later
/// construction failures are terminal read errors when the failed reader
is reached.
+ /// The positive preparation window bounds retained readers, including
in-flight builds,
+ /// independently of the build executor's parallelism.
static Result<std::unique_ptr<ConcatBatchReader>> CreateLazy(
size_t reader_count, ReaderFactory factory,
const std::shared_ptr<arrow::MemoryPool>& arrow_pool,
- const std::shared_ptr<Executor>& build_executor =
GetReaderBuildExecutor());
+ const std::shared_ptr<Executor>& build_executor =
GetReaderBuildExecutor(),
+ int32_t preparation_window = kDefaultPreparationWindow);
Review Comment:
Removed the default executor argument from `CreateLazy` and updated its
callers to pass the executor explicitly. `preparation_window` now comes before
`arrow_pool` and `build_executor`, keeping the pool and executor at the end.
[Updated
signature](https://github.com/apache/paimon-cpp/blob/ebe1e01b34628cff5c3893719e4b0af889f147d9/src/paimon/common/reader/concat_batch_reader.h#L56-L59)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]