alexeykudinkin commented on code in PR #5241:
URL: https://github.com/apache/hudi/pull/5241#discussion_r844303357
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataMergedLogRecordReader.java:
##########
@@ -64,7 +64,7 @@ private HoodieMetadataMergedLogRecordReader(FileSystem fs,
String basePath, Stri
ExternalSpillableMap.DiskMapType
diskMapType,
boolean
isBitCaskDiskMapCompressionEnabled,
Option<InstantRange>
instantRange, boolean enableFullScan) {
- super(fs, basePath, logFilePaths, readerSchema, latestInstantTime,
maxMemorySizeInBytes, false, false, bufferSize,
+ super(fs, basePath, logFilePaths, readerSchema, latestInstantTime,
maxMemorySizeInBytes, true, false, bufferSize,
Review Comment:
We should not couple those -- these configs control different aspects:
1. `forceFullScan` (renamed it to make its semantic crystal clear) -- forces
to read all records from the block
2. `readBlocksLazily` -- controls whether we merge blocks eagerly as we read
them or backward-pass when all of them are read.
We should remove `readBlocksLazily` altogether actually and leave only
backward-pass behavior, since the other alternatives will be yielding incorrect
merge results (one example is HUDI-3342, another example is incorrect handling
of deletes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
