yihua commented on a change in pull request #4974:
URL: https://github.com/apache/hudi/pull/4974#discussion_r828554297
##########
File path:
hudi-common/src/main/java/org/apache/hudi/common/util/CompactionUtils.java
##########
@@ -195,10 +196,76 @@ public static HoodieCompactionPlan
getCompactionPlan(HoodieTableMetaClient metaC
/**
* Return all pending compaction instant times.
- *
+ *
* @return
*/
public static List<HoodieInstant>
getPendingCompactionInstantTimes(HoodieTableMetaClient metaClient) {
return
metaClient.getActiveTimeline().filterPendingCompactionTimeline().getInstants().collect(Collectors.toList());
}
+
+ /**
+ * Returns a pair of (timeline containing the delta commits after the latest
completed
+ * compaction commit, the completed compaction commit instant), if the
latest completed
+ * compaction commit is present; a pair of (timeline containing all the
delta commits,
+ * the first delta commit instant), if there is no completed compaction
commit.
+ *
+ * @param activeTimeline Active timeline of a table.
+ * @return Pair of timeline containing delta commits and an instant.
+ */
+ public static Option<Pair<HoodieTimeline, HoodieInstant>>
getDeltaCommitsSinceLatestCompaction(
+ HoodieActiveTimeline activeTimeline) {
+ Option<HoodieInstant> lastCompaction = activeTimeline.getCommitTimeline()
+ .filterCompletedInstants().lastInstant();
+ HoodieTimeline deltaCommits = activeTimeline.getDeltaCommitTimeline();
+
+ HoodieInstant latestInstant;
+ if (lastCompaction.isPresent()) {
+ latestInstant = lastCompaction.get();
+ // timeline containing the delta commits after the latest completed
compaction commit,
+ // and the completed compaction commit instant
+ return Option.of(Pair.of(deltaCommits.findInstantsAfter(
+ latestInstant.getTimestamp(), Integer.MAX_VALUE),
lastCompaction.get()));
+ } else {
+ if (deltaCommits.countInstants() > 0) {
+ latestInstant = deltaCommits.firstInstant().get();
+ // timeline containing all the delta commits, and the first delta
commit instant
+ return Option.of(Pair.of(deltaCommits.findInstantsAfterOrEquals(
+ latestInstant.getTimestamp(), Integer.MAX_VALUE), latestInstant));
+ } else {
+ return Option.empty();
+ }
+ }
+ }
+
+ /**
+ * Gets the oldest instant to keep for MOR compaction.
+ * If there is no completed compaction,
+ * num delta commits >= "hoodie.compact.inline.max.delta.commits"
+ * If there is a completed compaction,
+ * num delta commits after latest completed compaction >=
"hoodie.compact.inline.max.delta.commits"
+ *
+ * @param activeTimeline Active timeline of a table.
+ * @param maxDeltaCommits Maximum number of delta commits that trigger the
compaction plan,
+ * i.e., "hoodie.compact.inline.max.delta.commits".
+ * @return the oldest instant to keep for MOR compaction.
+ */
+ public static Option<HoodieInstant> getOldestInstantToKeepForCompaction(
+ HoodieActiveTimeline activeTimeline, int maxDeltaCommits) {
Review comment:
This is specific to compaction. We need to retain the deltacommits in
the timeline for compaction to work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
