yihua commented on code in PR #8156:
URL: https://github.com/apache/hudi/pull/8156#discussion_r1133047376
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieMetadataTableValidator.java:
##########
@@ -533,7 +533,19 @@ private List<String>
validatePartitions(HoodieSparkEngineContext engineContext,
Option<String> instantOption =
hoodiePartitionMetadata.readPartitionCreatedCommitTime();
if (instantOption.isPresent()) {
String instantTime = instantOption.get();
- return completedTimeline.containsOrBeforeTimelineStarts(instantTime);
+ // There are two cases where the created commit time is written to the
partition metadata:
+ // (1) Commit C1 creates the partition and C1 succeeds, the partition
metadata has C1 as
+ // the created commit time.
+ // (2) Commit C1 creates the partition, the partition metadata is
written, and C1 fails
+ // during writing data files. Next time, C2 adds new data to the same
partition after C1
+ // is rolled back. In this case, the partition metadata still has C1
as the created commit
+ // time, since Hudi does not rewrite the partition metadata in C2.
+ if (!completedTimeline.containsOrBeforeTimelineStarts(instantTime)) {
+ Option<HoodieInstant> lastInstant = completedTimeline.lastInstant();
+ return lastInstant.isPresent()
+ && lastInstant.get().getTimestamp().compareTo(instantTime) >= 0;
Review Comment:
Addressed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
