hddong commented on a change in pull request #1554:
URL: https://github.com/apache/incubator-hudi/pull/1554#discussion_r421387588
##########
File path:
hudi-cli/src/main/java/org/apache/hudi/cli/commands/RepairsCommand.java
##########
@@ -64,19 +69,35 @@ public String deduplicate(
@CliOption(key = {"repairedOutputPath"}, help = "Location to place the
repaired files",
mandatory = true) final String repairedOutputPath,
@CliOption(key = {"sparkProperties"}, help = "Spark Properties File
Path",
- mandatory = true) final String sparkPropertiesPath)
+ unspecifiedDefaultValue = "") String sparkPropertiesPath,
+ @CliOption(key = "sparkMaster", unspecifiedDefaultValue = "", help =
"Spark Master ") String master,
+ @CliOption(key = "sparkMemory", unspecifiedDefaultValue = "4G",
+ help = "Spark executor memory") final String sparkMemory,
+ @CliOption(key = {"dryrun"},
+ help = "Should we actually remove duplicates or just run and store
result to repairedOutputPath",
+ unspecifiedDefaultValue = "true") final boolean dryRun)
throws Exception {
+ if (StringUtils.isNullOrEmpty(sparkPropertiesPath)) {
+ sparkPropertiesPath =
+
Utils.getDefaultPropertiesFile(JavaConverters.mapAsScalaMapConverter(System.getenv()).asScala());
+ }
+
SparkLauncher sparkLauncher = SparkUtil.initLauncher(sparkPropertiesPath);
- sparkLauncher.addAppArgs(SparkMain.SparkCommand.DEDUPLICATE.toString(),
duplicatedPartitionPath, repairedOutputPath,
- HoodieCLI.getTableMetaClient().getBasePath());
+ sparkLauncher.addAppArgs(SparkMain.SparkCommand.DEDUPLICATE.toString(),
master, sparkMemory,
Review comment:
> The same suggestion, we should try to define a data structure? We can
refactor it later.
We can focus on PR(#1174), but it was left behind for too long.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
