advancedxy commented on code in PR #495:
URL: https://github.com/apache/incubator-uniffle/pull/495#discussion_r1095294888
##########
client-spark/common/src/test/java/org/apache/spark/shuffle/writer/WriteBufferManagerTest.java:
##########
@@ -41,14 +40,20 @@
public class WriteBufferManagerTest {
+ static {
+ // trigger spark config package initialization before RssSparkConfig to
avoid key, spark.shuffle.compress,
+ // duplication in spark3
+ org.apache.spark.internal.config.package$.MODULE$.EXECUTOR_MEMORY();
Review Comment:
> Uniffle is shuffle plugin which gets startup after Spark internal config.
As tested, there is no issue in real spark job. And the UT tests with spark2
and spark3 also passed without above code.
Thanks for the explanation. It should be ok for most cases. In some
rare/unusual case, it's still possible to get duplicated:
```
new RssSparkConf(); // to check some default configurations
// do some calculations.
new SparkContext with RssShuffleManager enabled;
```
> Let me just add string field to make thing easier.
🙏
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
