satishkotha commented on a change in pull request #2678:
URL: https://github.com/apache/hudi/pull/2678#discussion_r594753875
##########
File path:
hudi-cli/src/main/java/org/apache/hudi/cli/commands/CommitsCommand.java
##########
@@ -431,4 +442,20 @@ public String syncCommits(@CliOption(key = {"path"}, help
= "Path of the table t
return "Load sync state between " +
HoodieCLI.getTableMetaClient().getTableConfig().getTableName() + " and "
+ HoodieCLI.syncTableMetadata.getTableConfig().getTableName();
}
+
+ /*
+ Checks whether a commit or replacecommit action exists in the timeline.
+ * */
+ private Option<HoodieInstant> getCommitOrReplaceCommitInstant(HoodieTimeline
timeline, String instantTime) {
Review comment:
consider changing signature to return Option<HoodieCommitMetadata> and
deserialize instant details inside this method. This would avoid repetition to
get instant details in multiple places. You can also do additional validation.
for example: for replace commit, deserialize using HoodieReplaceCommitMetadata
class
##########
File path:
hudi-cli/src/test/java/org/apache/hudi/cli/testutils/HoodieTestReplaceCommitMetadatGenerator.java
##########
@@ -0,0 +1,74 @@
+package org.apache.hudi.cli.testutils;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hudi.common.model.HoodieReplaceCommitMetadata;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.testutils.FileCreateUtils;
+import org.apache.hudi.common.util.Option;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.UUID;
+
+import static org.apache.hudi.common.testutils.FileCreateUtils.baseFileName;
+import static org.apache.hudi.common.util.CollectionUtils.createImmutableList;
+
+public class HoodieTestReplaceCommitMetadatGenerator extends
HoodieTestCommitMetadataGenerator{
+ public static void createReplaceCommitFileWithMetadata(String basePath,
String commitTime, Configuration configuration,
+ Option<Integer>
writes, Option<Integer> updates) throws Exception {
+ createReplaceCommitFileWithMetadata(basePath, commitTime,
configuration, UUID.randomUUID().toString(),
+ UUID.randomUUID().toString(), writes, updates);
+ }
+
+ private static void createReplaceCommitFileWithMetadata(String basePath,
String commitTime, Configuration configuration,
+ String fileId1,
String fileId2, Option<Integer> writes,
+ Option<Integer>
updates) throws Exception {
+ List<String> commitFileNames =
Arrays.asList(HoodieTimeline.makeCommitFileName(commitTime),
Review comment:
Can we reuse replace commit generator from other places? HoodieTestTable
for example?
##########
File path:
hudi-cli/src/main/java/org/apache/hudi/cli/commands/CommitsCommand.java
##########
@@ -266,12 +267,15 @@ public String showCommitPartitions(
HoodieActiveTimeline activeTimeline =
HoodieCLI.getTableMetaClient().getActiveTimeline();
HoodieTimeline timeline =
activeTimeline.getCommitsTimeline().filterCompletedInstants();
- HoodieInstant commitInstant = new HoodieInstant(false,
HoodieTimeline.COMMIT_ACTION, instantTime);
- if (!timeline.containsInstant(commitInstant)) {
+ Option<HoodieInstant> hoodieInstantOptional =
getCommitOrReplaceCommitInstant(timeline, instantTime);
+ if (!hoodieInstantOptional.isPresent()) {
return "Commit " + instantTime + " not found in Commits " + timeline;
}
- HoodieCommitMetadata meta =
HoodieCommitMetadata.fromBytes(activeTimeline.getInstantDetails(commitInstant).get(),
+
+ HoodieInstant hoodieInstant = hoodieInstantOptional.get();
+
+ HoodieCommitMetadata meta =
HoodieCommitMetadata.fromBytes(activeTimeline.getInstantDetails(hoodieInstant).get(),
HoodieCommitMetadata.class);
List<Comparable[]> rows = new ArrayList<>();
for (Map.Entry<String, List<HoodieWriteStat>> entry :
meta.getPartitionToWriteStats().entrySet()) {
Review comment:
it'd be nice to compute totalfFilesReplaced and show it in the table.
It could be 0 for regular commits.
##########
File path:
hudi-cli/src/main/java/org/apache/hudi/cli/commands/CommitsCommand.java
##########
@@ -431,4 +442,20 @@ public String syncCommits(@CliOption(key = {"path"}, help
= "Path of the table t
return "Load sync state between " +
HoodieCLI.getTableMetaClient().getTableConfig().getTableName() + " and "
+ HoodieCLI.syncTableMetadata.getTableConfig().getTableName();
}
+
+ /*
+ Checks whether a commit or replacecommit action exists in the timeline.
+ * */
+ private Option<HoodieInstant> getCommitOrReplaceCommitInstant(HoodieTimeline
timeline, String instantTime) {
+ HoodieInstant hoodieInstant = new HoodieInstant(false,
HoodieTimeline.COMMIT_ACTION, instantTime);
+
+ if (!timeline.containsInstant(hoodieInstant)) {
+ hoodieInstant = new HoodieInstant(false,
HoodieTimeline.REPLACE_COMMIT_ACTION, instantTime);
+ if (!timeline.containsInstant(hoodieInstant)) {
+ return Option.empty();
Review comment:
should we also include DELTA COMMIT here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
