(celeborn) branch branch-0.5 updated: [CELEBORN-775][FOLLOWUP] Fix executorCores calculation in SparkShuffleManager for Spark2 local mode

[nicholasjiang]

This is an automated email from the ASF dual-hosted git repository.

nicholasjiang pushed a commit to branch branch-0.5
in repository https://gitbox.apache.org/repos/asf/celeborn.git


The following commit(s) were added to refs/heads/branch-0.5 by this push:
     new 1b04ed01d [CELEBORN-775][FOLLOWUP] Fix executorCores calculation in 
SparkShuffleManager for Spark2 local mode
1b04ed01d is described below

commit 1b04ed01d4017489b47165eb4ada71e89e8eac4c
Author: sychen <syc...@ctrip.com>
AuthorDate: Thu Jun 13 17:39:30 2024 +0800

    [CELEBORN-775][FOLLOWUP] Fix executorCores calculation in 
SparkShuffleManager for Spark2 local mode
    
    ### What changes were proposed in this pull request?
    
    ### Why are the changes needed?
    CELEBORN-775 The `executorCores` method was introduced in Spark2, but it 
was not used.
    
    ### Does this PR introduce _any_ user-facing change?
    No
    
    ### How was this patch tested?
    GA
    
    Closes #2562 from cxzl25/CELEBORN-775-FOLLOWUP.
    
    Authored-by: sychen <syc...@ctrip.com>
    Signed-off-by: SteNicholas <programg...@163.com>
    (cherry picked from commit e177a20803031257e49f0225ef79cb096704593f)
    Signed-off-by: SteNicholas <programg...@163.com>
---
 .../java/org/apache/spark/shuffle/celeborn/SparkShuffleManager.java     | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git 
a/client-spark/spark-2/src/main/java/org/apache/spark/shuffle/celeborn/SparkShuffleManager.java
 
b/client-spark/spark-2/src/main/java/org/apache/spark/shuffle/celeborn/SparkShuffleManager.java
index 1edd27cd4..6aa63ec80 100644
--- 
a/client-spark/spark-2/src/main/java/org/apache/spark/shuffle/celeborn/SparkShuffleManager.java
+++ 
b/client-spark/spark-2/src/main/java/org/apache/spark/shuffle/celeborn/SparkShuffleManager.java
@@ -69,7 +69,7 @@ public class SparkShuffleManager implements ShuffleManager {
     this.conf = conf;
     this.isDriver = isDriver;
     this.celebornConf = SparkUtils.fromSparkConf(conf);
-    this.cores = conf.getInt(SparkLauncher.EXECUTOR_CORES, 1);
+    this.cores = executorCores(conf);
     this.fallbackPolicyRunner = new 
CelebornShuffleFallbackPolicyRunner(celebornConf);
     this.sendBufferPoolCheckInterval = 
celebornConf.clientPushSendBufferPoolExpireCheckInterval();
     this.sendBufferPoolExpireTimeout = 
celebornConf.clientPushSendBufferPoolExpireTimeout();