[incubator-uniffle] branch master updated: [#630] feat(client): Disable the localShuffleReader by default in Spark3 client (#636)

[zuston]

This is an automated email from the ASF dual-hosted git repository.

zuston pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-uniffle.git


The following commit(s) were added to refs/heads/master by this push:
     new 85b7b3b9 [#630] feat(client): Disable the localShuffleReader by 
default in Spark3 client (#636)
85b7b3b9 is described below

commit 85b7b3b95539bc23f0ca0ea594539d1c5b40c614
Author: Junfan Zhang <zus...@apache.org>
AuthorDate: Tue Feb 21 19:14:27 2023 +0800

    [#630] feat(client): Disable the localShuffleReader by default in Spark3 
client (#636)
    
    ### What changes were proposed in this pull request?
    
    Disable the localShuffleReader by default in Spark3 client
    
    ### Why are the changes needed?
    
    in `RssShuffleManager`, the local shuffle reader should be disabled by 
default (`spark.sql.adaptive.localShuffleReader.enabled=false`), as it would 
cause too many random small IOs and network connections with shuffle servers.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes
    
    ### How was this patch tested?
    
    Don't need
---
 .../src/main/java/org/apache/spark/shuffle/RssShuffleManager.java    | 2 ++
 docs/client_guide.md                                                 | 5 +++++
 2 files changed, 7 insertions(+)

diff --git 
a/client-spark/spark3/src/main/java/org/apache/spark/shuffle/RssShuffleManager.java
 
b/client-spark/spark3/src/main/java/org/apache/spark/shuffle/RssShuffleManager.java
index a768a449..8d17c093 100644
--- 
a/client-spark/spark3/src/main/java/org/apache/spark/shuffle/RssShuffleManager.java
+++ 
b/client-spark/spark3/src/main/java/org/apache/spark/shuffle/RssShuffleManager.java
@@ -201,6 +201,8 @@ public class RssShuffleManager implements ShuffleManager {
     // External shuffle service is not supported when using remote shuffle 
service
     sparkConf.set("spark.shuffle.service.enabled", "false");
     LOG.info("Disable external shuffle service in RssShuffleManager.");
+    sparkConf.set("spark.sql.adaptive.localShuffleReader.enabled", "false");
+    LOG.info("Disable local shuffle reader in RssShuffleManager.");
     taskToSuccessBlockIds = Maps.newConcurrentMap();
     taskToFailedBlockIds = Maps.newConcurrentMap();
     // for non-driver executor, start a thread for sending shuffle data to 
shuffle server
diff --git a/docs/client_guide.md b/docs/client_guide.md
index 12c99c91..8b1b008e 100644
--- a/docs/client_guide.md
+++ b/docs/client_guide.md
@@ -80,6 +80,11 @@ and Continuous partition assignment mechanism.
         # Default value is 1.0, used to estimate task concurrency, how likely 
is this part of the resource between spark.dynamicAllocation.minExecutors and 
spark.dynamicAllocation.maxExecutors to be allocated
         --conf spark.rss.estimate.task.concurrency.dynamic.factor=1.0
         ```
+   
+Since v0.8.0, `RssShuffleManager` would disable local shuffle reader(`set 
spark.sql.adaptive.localShuffleReader.enabled=false`) optimization by default.
+
+Local shuffle reader as its name indicates is suitable and optimized for 
spark's external shuffle service, and shall not be used for remote shuffle 
service. It would cause many random small IOs and network connections with 
Uniffle's shuffle server
+
 ### Deploy MapReduce Client Plugin
 
 1. Add client jar to the classpath of each NodeManager, e.g., 
<HADOOP>/share/hadoop/mapreduce/