Neuw84 opened a new issue, #5456: URL: https://github.com/apache/hudi/issues/5456
**Describe the problem you faced** When trying to run HoodieDeltaStreamer on AWS Glue I found that the Spark master has no option to inherit from the environment as it defaults to ```local[2]```. In these kind of Serverless environments where you do not have access to the master this configuration should be inherited This can be seen on line 329 on [HoodieDeltaStreamer](https://github.com/apache/hudi/blob/master/hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieDeltaStreamer.java). ``` public String sparkMaster = "local[2]"; ``` This should be changed for supporting this kind of scenarios, a JavaSparkContext option where no Spark master is defined should be there. **Expected behavior** The Spark master shouldn't have a default as there are some environments (usually serverless such as AWS Glue) where it will be inherited. **Environment Description** * Hudi version : 0.9.0 * Spark version : Spark 3.1.1 * Storage (HDFS/S3/GCS..) : S3 * Running on Docker? (yes/no) : no **Additional context** If required I think I could work on this as I have quite good Java experience. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
