yihua commented on code in PR #10102:
URL: https://github.com/apache/hudi/pull/10102#discussion_r1408590210
##########
hudi-common/src/main/java/org/apache/hudi/common/table/log/BaseHoodieLogRecordReader.java:
##########
@@ -260,7 +260,7 @@ private void scanInternalV1(Option<KeySpec> keySpecOpt) {
&&
!HoodieTimeline.compareTimestamps(logBlock.getLogBlockHeader().get(INSTANT_TIME),
HoodieTimeline.LESSER_THAN_OR_EQUALS, this.latestInstantTime
)) {
// hit a block with instant time greater than should be processed,
stop processing further
- break;
+ continue;
}
Review Comment:
The `BaseHoodieLogRecordReader` is used by the new file group reader only
and this one uses `HoodieLogFormatReverseReader` which reads the log files in
reverse order based on the input log file list. Right now for time travel
queries, the file system view still returns more logs than required (e.g., bf,
log_t1, log_t2, log_t3, log_t4, for time travel query with `as of t2`), so this
change makes sure the processing does not stop unnecessarily before the file
system view returns the exact files to process.
@linliu-code could you add this detail to the other ticket fixing the file
system view APIs based on completion time, so we should revisit this part
alongside the FSV fix?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
