danny0405 commented on code in PR #10791:
URL: https://github.com/apache/hudi/pull/10791#discussion_r1508670977
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/compact/ScheduleCompactionActionExecutor.java:
##########
@@ -93,11 +93,12 @@ public Option<HoodieCompactionPlan> execute() {
// TODO(yihua): this validation is removed for Java client used by
kafka-connect. Need to revisit this.
if (config.getEngineType() == EngineType.SPARK) {
// if there are inflight writes, their instantTime must not be less
than that of compaction instant time
-
table.getActiveTimeline().getCommitsTimeline().filterPendingExcludingMajorAndMinorCompaction().firstInstant()
- .ifPresent(earliestInflight -> ValidationUtils.checkArgument(
-
HoodieTimeline.compareTimestamps(earliestInflight.getTimestamp(),
HoodieTimeline.GREATER_THAN, instantTime),
- "Earliest write inflight instant time must be later than
compaction time. Earliest :" + earliestInflight
- + ", Compaction scheduled at " + instantTime));
+ Option<HoodieInstant> earliestInflightOpt =
table.getActiveTimeline().getCommitsTimeline().filterPendingExcludingMajorAndMinorCompaction().firstInstant();
+ if (earliestInflightOpt.isPresent() &&
!HoodieTimeline.compareTimestamps(earliestInflightOpt.get().getTimestamp(),
HoodieTimeline.GREATER_THAN, instantTime)) {
+ LOG.warn("Earliest write inflight instant time must be later than
compaction time. Earliest :" + earliestInflightOpt.get()
+ + ", Compaction scheduled at " + instantTime + ". Hence skipping
to schedule compaction");
+ return Option.empty();
Review Comment:
Makes sense somehow, we can actually remove this limination because we
migrate to completion time based file slicing for release 1.0, the scheduler
will filter out the files with the corespondent files.
For 0.x release, this is a usability improvement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
