ksmou commented on code in PR #9229:
URL: https://github.com/apache/hudi/pull/9229#discussion_r1272927559
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieCompactor.java:
##########
@@ -101,6 +104,12 @@ public static class Config implements Serializable {
public String runningMode = null;
@Parameter(names = {"--strategy", "-st"}, description = "Strategy Class",
required = false)
public String strategyClassName =
LogFileSizeBasedCompactionStrategy.class.getName();
+ @Parameter(names = {"--job-max-processing-time-ms", "-mt"}, description =
"Take effect when using --mode/-m execute or scheduleAndExecute. "
+ + "If maxProcessingTimeMs passed but compaction job is still
unfinished, hoodie would consider this job as failed and relaunch.")
+ public long maxProcessingTimeMs = 0;
+ @Parameter(names = {"--retry-last-failed-compaction-job", "-rc"},
description = "Take effect when using --mode/-m execute or scheduleAndExecute. "
Review Comment:
Yes. We just only choose a failed compaction instant here. It will been
rolled back in subsequent executions `SparkRDDTableServiceClient#compact(String
compactionInstantTime, boolean shouldComplete)`
```java
// org/apache/hudi/client/SparkRDDTableServiceClient.java
protected HoodieWriteMetadata<JavaRDD<WriteStatus>> compact(String
compactionInstantTime, boolean shouldComplete) {
HoodieSparkTable<T> table = HoodieSparkTable.create(config, context);
HoodieTimeline pendingCompactionTimeline =
table.getActiveTimeline().filterPendingCompactionTimeline();
HoodieInstant inflightInstant =
HoodieTimeline.getCompactionInflightInstant(compactionInstantTime);
if (pendingCompactionTimeline.containsInstant(inflightInstant)) {
table.rollbackInflightCompaction(inflightInstant, commitToRollback ->
getPendingRollbackInfo(table.getMetaClient(), commitToRollback, false));
table.getMetaClient().reloadActiveTimeline();
}
compactionTimer = metrics.getCompactionCtx();
HoodieWriteMetadata<HoodieData<WriteStatus>> writeMetadata =
table.compact(context, compactionInstantTime);
HoodieWriteMetadata<JavaRDD<WriteStatus>> compactionMetadata =
writeMetadata.clone(HoodieJavaRDD.getJavaRDD(writeMetadata.getWriteStatuses()));
if (shouldComplete &&
compactionMetadata.getCommitMetadata().isPresent()) {
completeTableService(TableServiceType.COMPACT,
compactionMetadata.getCommitMetadata().get(), table, compactionInstantTime,
Option.ofNullable(HoodieJavaRDD.of(compactionMetadata.getWriteStatuses())));
}
return compactionMetadata;
}
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
