Neuw84 commented on code in PR #5630:
URL: https://github.com/apache/hudi/pull/5630#discussion_r969215537
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/UtilHelpers.java:
##########
@@ -285,6 +285,24 @@ private static SparkConf buildSparkConf(String appName,
String defaultMaster, Ma
return SparkRDDWriteClient.registerClasses(sparkConf);
}
+ private static SparkConf buildSparkConf(String appName, Map<String, String>
additionalConfigs) {
+ final SparkConf sparkConf = new SparkConf().setAppName(appName);
+ sparkConf.set("spark.ui.port", "8090");
+ sparkConf.setIfMissing("spark.driver.maxResultSize", "2g");
+ sparkConf.set("spark.serializer",
"org.apache.spark.serializer.KryoSerializer");
+ sparkConf.set("spark.hadoop.mapred.output.compress", "true");
+ sparkConf.set("spark.hadoop.mapred.output.compression.codec", "true");
+ sparkConf.set("spark.hadoop.mapred.output.compression.codec",
"org.apache.hadoop.io.compress.GzipCodec");
+ sparkConf.set("spark.hadoop.mapred.output.compression.type", "BLOCK");
Review Comment:
Hi @yihua,
I just copied them from the other constructor. Most of them won´t have
effect on Glue as the change of internal configs is very restricted.
Will try to make that happen, I was thinking on posting on the AWS blog but
it will good to post this also on Hudi website/docs.
Will work on the Hudi website next.
Thanks for your inputs!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
