(incubator-celeborn) branch main updated: [CELEBORN-1209][FOLLOW] Fix warning based on Spark Enabled ShuffleTracking after Spark 3.4

[chengpan]

This is an automated email from the ASF dual-hosted git repository.

chengpan pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-celeborn.git


The following commit(s) were added to refs/heads/main by this push:
     new 865f6f3d8 [CELEBORN-1209][FOLLOW] Fix warning based on Spark Enabled 
ShuffleTracking after Spark 3.4
865f6f3d8 is described below

commit 865f6f3d83347386c894b0fd45126fbddf2bb4b5
Author: zwangsheng <binjiey...@apache.org>
AuthorDate: Fri Jan 5 20:01:54 2024 +0800

    [CELEBORN-1209][FOLLOW] Fix warning based on Spark Enabled ShuffleTracking 
after Spark 3.4
    
    ### What changes were proposed in this pull request?
    
    [SPARK-39846](https://issues.apache.org/jira/browse/SPARK-39846) set 
shuffle tracking enabled default value to true, fix warning.
    
    ### Why are the changes needed?
    As title
    
    ### Does this PR introduce _any_ user-facing change?
    Yes
    
    ### How was this patch tested?
    
    Closes #2204 from zwangsheng/CELEBORN-1209-1.
    
    Authored-by: zwangsheng <binjiey...@apache.org>
    Signed-off-by: Cheng Pan <cheng...@apache.org>
---
 .../apache/spark/shuffle/celeborn/SparkShuffleManager.java  | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)

diff --git 
a/client-spark/spark-3/src/main/java/org/apache/spark/shuffle/celeborn/SparkShuffleManager.java
 
b/client-spark/spark-3/src/main/java/org/apache/spark/shuffle/celeborn/SparkShuffleManager.java
index 264bf7766..35d847f46 100644
--- 
a/client-spark/spark-3/src/main/java/org/apache/spark/shuffle/celeborn/SparkShuffleManager.java
+++ 
b/client-spark/spark-3/src/main/java/org/apache/spark/shuffle/celeborn/SparkShuffleManager.java
@@ -94,11 +94,16 @@ public class SparkShuffleManager implements ShuffleManager {
               + "use Celeborn as Remote Shuffle Service to avoid performance 
degradation.",
           SQLConf.LOCAL_SHUFFLE_READER_ENABLED().key());
     }
-    if (conf.getBoolean("spark.dynamicAllocation.shuffleTracking.enabled", 
false)) {
+    if ((Boolean) 
conf.get(package$.MODULE$.DYN_ALLOCATION_SHUFFLE_TRACKING_ENABLED())) {
       logger.warn(
-          "Detected spark.dynamicAllocation.shuffleTracking.enabled (default 
is false) is enabled, "
-              + "it's highly recommended to disable it when use Celeborn as 
Remote Shuffle Service "
-              + "to avoid performance degradation.");
+          "Detected {} (default is {}) is enabled, it's highly recommended to 
disable it when "
+              + "use Celeborn as Remote Shuffle Service to avoid performance 
degradation.",
+          package$.MODULE$.DYN_ALLOCATION_SHUFFLE_TRACKING_ENABLED().key(),
+          package$
+              .MODULE$
+              .DYN_ALLOCATION_SHUFFLE_TRACKING_ENABLED()
+              .defaultValue()
+              .map(v -> (Boolean) v));
     }
     this.conf = conf;
     this.isDriver = isDriver;