Wenning Ding created HUDI-3450:
----------------------------------
Summary: Avoid passing empty string spark master to hudi cli
Key: HUDI-3450
URL: https://issues.apache.org/jira/browse/HUDI-3450
Project: Apache Hudi
Issue Type: Bug
Reporter: Wenning Ding
When using Hudi CLI, when not passing SparkMaster, by default Hudi CLI should
use
[SparkUtil.DEFAULT_SPARK_MASTER|https://github.com/apache/hudi/blob/release-0.10.0/hudi-cli/src/main/java/org/apache/hudi/cli/utils/SparkUtil.java#L44].
However, w/ a recent [code
change|https://github.com/apache/hudi/commit/445208a0d20b457daeeb5f70995302c92dd19f31]
in OSS, when SparkMaster is not passed, it would set Spark master to {{""}}
which causes the following exception when initializing a Hudi CLI job:
{{org.apache.spark.SparkException: Could not parse Master URL: ''at
org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2999)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:567)
at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
at org.apache.hudi.cli.utils.SparkUtil.initJavaSparkConf(SparkUtil.java:115)
at org.apache.hudi.cli.utils.SparkUtil.initJavaSparkConf(SparkUtil.java:110)
at org.apache.hudi.cli.commands.SparkMain.main(SparkMain.java:88)}}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
