gripleaf commented on code in PR #243:
URL: https://github.com/apache/paimon-cpp/pull/243#discussion_r3843270013
##########
src/paimon/core/operation/abstract_split_read.cpp:
##########
@@ -152,13 +153,17 @@ Result<std::unique_ptr<ReaderBuilder>>
AbstractSplitRead::PrepareReaderBuilder(
Result<std::unique_ptr<FileBatchReader>>
AbstractSplitRead::CreateFileBatchReader(
const std::string& file_format_identifier, const std::string&
data_file_path,
- int64_t data_file_size, const ReaderBuilder* reader_builder) const {
+ int64_t data_file_size, std::unique_ptr<ReaderBuilder> reader_builder)
const {
if (context_->EnablePrefetch() && file_format_identifier != "blob" &&
file_format_identifier != "avro") {
+ if (context_->EnableLateMaterializing()) {
Review Comment:
Could we also support late materialization when prefetch is disabled?
Our main read path runs with EnablePrefetch(false). With the current
integration, LateMaterializingReaderBuilder is only installed inside the
prefetch branch, so those reads always fall back to the regular format reader
and never benefit from late materialization.
Late materialization and prefetch should ideally be independent
capabilities: the non-prefetch path should also wrap a compatible format
reader, or the two-phase orchestration should be moved above the
prefetch/non-prefetch split. PR #196 supports both modes by implementing the
probe/payload orchestration at the RawFileSplitRead layer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
