danny0405 commented on code in PR #9879:
URL: https://github.com/apache/hudi/pull/9879#discussion_r1363638292
##########
hudi-common/src/main/java/org/apache/hudi/common/table/log/AbstractHoodieLogRecordReader.java:
##########
@@ -241,7 +241,12 @@ private void scanInternalV1(Option<KeySpec> keySpecOpt) {
try {
// Iterate over the paths
logFormatReaderWrapper = new HoodieLogFormatReader(fs,
- logFilePaths.stream().map(logFile -> new HoodieLogFile(new
CachingPath(logFile))).collect(Collectors.toList()),
+ logFilePaths.stream()
+ .map(filePath -> new HoodieLogFile(new CachingPath(filePath)))
+ // hit an uncommitted file possibly from a failed write, skip
processing this one
+ .filter(logFile ->
completedInstantsTimeline.containsOrBeforeTimelineStarts(logFile.getDeltaCommitTime())
Review Comment:
There are some issues I can think of:
1. if the table has been upgraded, the table version check would become
invalid;
2. if there is coexistence of old and new log files, we need to filter out
only the new log files, but for old log files, we do not have very good manner
to check its base instant time because that might already be archived.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
