codope commented on code in PR #11864:
URL: https://github.com/apache/hudi/pull/11864#discussion_r1745798474
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieMetadataTableValidator.java:
##########
@@ -783,6 +788,17 @@ private void validateAllFileGroups(
.flatMap(HoodieFileGroup::getAllFileSlices).sorted(new
FileSliceComparator())
.collect(Collectors.toList());
}
+ if (!StringUtils.isNullOrEmpty(cfg.validateLastNFileSlices)) {
+ int lastNFileSlices = Integer.parseInt(cfg.validateLastNFileSlices);
+ int allFileSlicesFromFSCount = allFileSlicesFromFS.size();
+ if (allFileSlicesFromFSCount > lastNFileSlices) {
+ allFileSlicesFromFS =
allFileSlicesFromFS.subList(allFileSlicesFromFSCount - lastNFileSlices,
allFileSlicesFromFSCount);
Review Comment:
allFielSlicesFromFS is already sorted by commit time right?
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieMetadataTableValidator.java:
##########
@@ -283,6 +284,9 @@ public static class Config implements Serializable {
@Parameter(names = {"--validate-all-file-groups"}, description = "Validate
all file groups, and all file slices within file groups.", required = false)
public boolean validateAllFileGroups = false;
+ @Parameter(names = {"--validate-last-n-file-slices"}, description =
"Validate just the last N file slices for all file groups", required = false)
+ public String validateLastNFileSlices = null;
Review Comment:
Just to clarify usage, maybe we can say - `Validate just the last N file
slices for all file groups. Specify N.` Also, why not declare as Integer,
instad of doing String and parsing as int later?
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieMetadataTableValidator.java:
##########
@@ -1136,6 +1152,11 @@ private void validateFileSlices(
mismatch = true;
break;
}
+ // test for log files equality
+ if
(!fileSlice1.getLogFiles().collect(Collectors.toList()).equals(fileSlice2.getLogFiles().collect(Collectors.toList())))
{
Review Comment:
do we need to sort the list before comparing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
