xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r733521589
##########
File path:
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieRealtimeInputFormatUtils.java
##########
@@ -161,6 +162,46 @@
return rtSplits.toArray(new InputSplit[0]);
}
+ // pick all incremental files and add them to rtSplits then filter out those
files.
+ private static Map<Path, List<FileSplit>> filterOutIncrementalSplits(
+ List<FileSplit> fileSplitList,
Review comment:
The key point is not the number of commits， the key point is that at
commit3 file3 replace the file1 and file2。 we can set startcommit a little
smller than commit2 and max commit is 1.
set hoodie.testlog.consume.max.commits=1
set hoodie.testlog.consume.start.timestamp=commit2-1 then commit2 will be
picked
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
