(celeborn) branch main updated: [CELEBORN-1519] Do not update estimated partition size if it is unchanged

[chengpan]

This is an automated email from the ASF dual-hosted git repository.

chengpan pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/celeborn.git


The following commit(s) were added to refs/heads/main by this push:
     new afe40e6bb [CELEBORN-1519] Do not update estimated partition size if it 
is unchanged
afe40e6bb is described below

commit afe40e6bb06a6c3eb35dca568c6b71e4bd499a27
Author: Sanskar Modi <sanskarmod...@gmail.com>
AuthorDate: Thu Jul 25 16:25:40 2024 +0800

    [CELEBORN-1519] Do not update estimated partition size if it is unchanged
    
    ### What changes were proposed in this pull request?
    
    We will not update the estimated partition size if it is unchanged.
    
    ### Why are the changes needed?
    
    Celeborn currently triggers an workerinfo update even-though the estimated 
partition size is not changed. This leads to unnecessary logging and redundant 
worker info update operations.
    
    Example log -
    
    ```
    [master-partition-size-updater] WARN 
org.apache.celeborn.service.deploy.master.clustermeta.AbstractMetaManager - 
Celeborn cluster estimated partition size changed from 64.0 MiB to 64.0 MiB
    ```
    
    ### Does this PR introduce _any_ user-facing change?
    
    NO
    
    ### How was this patch tested?
    Existing UT's
    
    Closes #2642 from s0nskar/CELEBORN-1519.
    
    Authored-by: Sanskar Modi <sanskarmod...@gmail.com>
    Signed-off-by: Cheng Pan <cheng...@apache.org>
---
 .../service/deploy/master/clustermeta/AbstractMetaManager.java      | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git 
a/master/src/main/java/org/apache/celeborn/service/deploy/master/clustermeta/AbstractMetaManager.java
 
b/master/src/main/java/org/apache/celeborn/service/deploy/master/clustermeta/AbstractMetaManager.java
index 009f2743c..53f442091 100644
--- 
a/master/src/main/java/org/apache/celeborn/service/deploy/master/clustermeta/AbstractMetaManager.java
+++ 
b/master/src/main/java/org/apache/celeborn/service/deploy/master/clustermeta/AbstractMetaManager.java
@@ -472,6 +472,12 @@ public abstract class AbstractMetaManager implements 
IMetadataHandler {
     } else {
       estimatedPartitionSize = initialEstimatedPartitionSize;
     }
+
+    // Do not trigger update is estimated partition size value is unchanged
+    if (estimatedPartitionSize == oldEstimatedPartitionSize) {
+      return;
+    }
+
     LOG.warn(
         "Celeborn cluster estimated partition size changed from {} to {}",
         Utils.bytesToString(oldEstimatedPartitionSize),