[
https://issues.apache.org/jira/browse/HUDI-3994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17529535#comment-17529535
]
Angel Conde commented on HUDI-3994:
------------------------------------
Will provide a pull request of this.
> HoodieDeltaStreamer - Spark master shouldn't have a default
> -----------------------------------------------------------
>
> Key: HUDI-3994
> URL: https://issues.apache.org/jira/browse/HUDI-3994
> Project: Apache Hudi
> Issue Type: Improvement
> Components: deltastreamer
> Reporter: Angel Conde
> Priority: Trivial
> Labels: easyfix
> Fix For: 0.11.0
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> When trying to run HoodieDeltaStreamer on AWS Glue I found that the Spark
> master has no option to inherit from the environment as it defaults to
> {{{}local[2]{}}}. In these kind of Serverless environments where you do not
> have access to the master this configuration should be inherited
> This can be seen on line 329 on
> [HoodieDeltaStreamer|https://github.com/apache/hudi/blob/master/hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieDeltaStreamer.java].
> {{public String sparkMaster = "local[2]";}}
> This should be changed for supporting this kind of scenarios, a
> JavaSparkContext option where no Spark master is defined should be there.
> *Expected behavior*
> The Spark master shouldn't have a default as there are some environments
> (usually serverless such as AWS Glue) where it will be inherited.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
