yanghua commented on a change in pull request #1452: [HUDI-740]Fix can not
specify the sparkMaster and code clean for SparkUtil
URL: https://github.com/apache/incubator-hudi/pull/1452#discussion_r405272257
##########
File path: hudi-cli/src/main/java/org/apache/hudi/cli/utils/SparkUtil.java
##########
@@ -61,9 +62,14 @@ public static SparkLauncher initLauncher(String
propertiesFile) throws URISyntax
}
public static JavaSparkContext initJavaSparkConf(String name) {
+ return initJavaSparkConf(name, Option.empty(), Option.empty());
+ }
+
+ public static JavaSparkContext initJavaSparkConf(String name, Option<String>
master,
+ Option<String>
executorMemory) {
SparkConf sparkConf = new SparkConf().setAppName(name);
- String defMasterFromEnv = sparkConf.getenv("SPARK_MASTER");
+ String defMasterFromEnv = master.orElse(sparkConf.getenv("SPARK_MASTER"));
Review comment:
+1 to introduce `HoodieCliSparkConfig`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
