(celeborn) branch branch-0.5 updated: [CELEBORN-1522] Fix applicationId extraction from shuffle key

[feiwang]

This is an automated email from the ASF dual-hosted git repository.

feiwang pushed a commit to branch branch-0.5
in repository https://gitbox.apache.org/repos/asf/celeborn.git


The following commit(s) were added to refs/heads/branch-0.5 by this push:
     new 47a5d26b9 [CELEBORN-1522] Fix applicationId extraction from shuffle key
47a5d26b9 is described below

commit 47a5d26b93082b634d4a275ccc61dbabfeafad86
Author: Wang, Fei <fwan...@ebay.com>
AuthorDate: Wed Jul 24 21:50:09 2024 -0700

    [CELEBORN-1522] Fix applicationId extraction from shuffle key
    
    ### What changes were proposed in this pull request?
    
    Fix applicationId extraction from shuffle key.
    
    ### Why are the changes needed?
    
    For spark on k8s, the applicationId might be 
`spark-da4571bd2cbf491c892cbd4de40fc918`.
    
    Due the application extraction is not correct, the result of API 
`/api/v1/applications/top_disk_usages` is not correct.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No.
    
    ### How was this patch tested?
    
    Not needed, just leverage existing method.
    
    Closes #2645 from turboFei/fix_topdisk_usages.
    
    Authored-by: Wang, Fei <fwan...@ebay.com>
    Signed-off-by: Wang, Fei <fwan...@ebay.com>
    (cherry picked from commit e92d59ace5ac3aebc22022a6629d21b26ce10665)
    Signed-off-by: Wang, Fei <fwan...@ebay.com>
---
 .../celeborn/service/deploy/master/clustermeta/AbstractMetaManager.java | 2 +-
 .../apache/celeborn/service/deploy/worker/storage/StorageManager.scala  | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git 
a/master/src/main/java/org/apache/celeborn/service/deploy/master/clustermeta/AbstractMetaManager.java
 
b/master/src/main/java/org/apache/celeborn/service/deploy/master/clustermeta/AbstractMetaManager.java
index b9c1c2c67..6a34654ed 100644
--- 
a/master/src/main/java/org/apache/celeborn/service/deploy/master/clustermeta/AbstractMetaManager.java
+++ 
b/master/src/main/java/org/apache/celeborn/service/deploy/master/clustermeta/AbstractMetaManager.java
@@ -319,7 +319,7 @@ public abstract class AbstractMetaManager implements 
IMetadataHandler {
 
       registeredShuffle.forEach(
           shuffleKey -> {
-            String appId = shuffleKey.split("-")[0];
+            String appId = Utils.splitShuffleKey(shuffleKey)._1;
             if (!appHeartbeatTime.containsKey(appId)) {
               appHeartbeatTime.put(appId, System.currentTimeMillis());
             }
diff --git 
a/worker/src/main/scala/org/apache/celeborn/service/deploy/worker/storage/StorageManager.scala
 
b/worker/src/main/scala/org/apache/celeborn/service/deploy/worker/storage/StorageManager.scala
index e767060c7..7093de140 100644
--- 
a/worker/src/main/scala/org/apache/celeborn/service/deploy/worker/storage/StorageManager.scala
+++ 
b/worker/src/main/scala/org/apache/celeborn/service/deploy/worker/storage/StorageManager.scala
@@ -493,7 +493,7 @@ final private[worker] class StorageManager(conf: 
CelebornConf, workerSource: Abs
         keyedWriters._1 -> 
keyedWriters._2.values().asScala.map(_.getFileLength).sum
       }
     }.toList.map { case (shuffleKey, usage) =>
-      shuffleKey.split("-")(0) -> usage
+      Utils.splitShuffleKey(shuffleKey)._1 -> usage
     }.groupBy(_._1).map { case (key, values) =>
       key -> values.map(_._2).sum
     }.toSeq.sortBy(_._2).reverse.take(conf.metricsAppTopDiskUsageCount * 
2).toMap.asJava