Angel Conde created HUDI-3994:
----------------------------------
Summary: HoodieDeltaStreamer - Spark master shouldn't have a
default
Key: HUDI-3994
URL: https://issues.apache.org/jira/browse/HUDI-3994
Project: Apache Hudi
Issue Type: Improvement
Components: deltastreamer
Reporter: Angel Conde
Fix For: 0.11.0
When trying to run HoodieDeltaStreamer on AWS Glue I found that the Spark
master has no option to inherit from the environment as it defaults to
{{{}local[2]{}}}. In these kind of Serverless environments where you do not
have access to the master this configuration should be inherited
This can be seen on line 329 on
[HoodieDeltaStreamer|https://github.com/apache/hudi/blob/master/hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieDeltaStreamer.java].
{{public String sparkMaster = "local[2]";}}
This should be changed for supporting this kind of scenarios, a
JavaSparkContext option where no Spark master is defined should be there.
*Expected behavior*
The Spark master shouldn't have a default as there are some environments
(usually serverless such as AWS Glue) where it will be inherited.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
