(celeborn) branch branch-0.5 updated: [CELEBORN-1638] Improve the slots allocator performance

[rexxiong]

This is an automated email from the ASF dual-hosted git repository.

rexxiong pushed a commit to branch branch-0.5
in repository https://gitbox.apache.org/repos/asf/celeborn.git


The following commit(s) were added to refs/heads/branch-0.5 by this push:
     new cd18e4127 [CELEBORN-1638] Improve the slots allocator performance
cd18e4127 is described below

commit cd18e4127a57d3437a844f4da2a32ca635b5146c
Author: wankunde <wanku...@163.com>
AuthorDate: Fri Oct 11 11:24:08 2024 +0800

    [CELEBORN-1638] Improve the slots allocator performance
    
    ### What changes were proposed in this pull request?
    
    When we try to allocate a large number of slots, the iterator.move of 
ArrayList partitionIdList will be very slow。
    
    
![CPU_time](https://github.com/user-attachments/assets/e33fedbc-14bc-43fe-a3fb-21424cdc3032)
    
    We can use LinkedList instead of ArrayList to optimize slots allocator.
    The execution time dropped from 97 secs to 12 secs
    
    <img width="845" alt="image" 
src="https://github.com/user-attachments/assets/3053bb1c-830e-44d4-88c5-3cc017e91994";>
    
    ### Why are the changes needed?
    
    To improve the slot allocator performance
    
    ### Does this PR introduce _any_ user-facing change?
    
    No
    
    ### How was this patch tested?
    
    We can test this PR with 
`SlotsAllocatorRackAwareSuiteJ.testRackAwareRoundRobinReplicaDistribution()`
    
    Closes #2799 from wankunde/slots_allocator.
    
    Authored-by: wankunde <wanku...@163.com>
    Signed-off-by: Shuang <lvshuang....@alibaba-inc.com>
    (cherry picked from commit 9f31087097c46b2c90803b679c4c5ef22689ccb0)
    Signed-off-by: Shuang <lvshuang....@alibaba-inc.com>
---
 .../java/org/apache/celeborn/service/deploy/master/SlotsAllocator.java  | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git 
a/master/src/main/java/org/apache/celeborn/service/deploy/master/SlotsAllocator.java
 
b/master/src/main/java/org/apache/celeborn/service/deploy/master/SlotsAllocator.java
index ef42c9992..1c0237b43 100644
--- 
a/master/src/main/java/org/apache/celeborn/service/deploy/master/SlotsAllocator.java
+++ 
b/master/src/main/java/org/apache/celeborn/service/deploy/master/SlotsAllocator.java
@@ -321,7 +321,7 @@ public class SlotsAllocator {
     // workerInfo -> (diskIndexForPrimary, diskIndexForReplica)
     Map<WorkerInfo, Integer> workerDiskIndexForPrimary = new HashMap<>();
     Map<WorkerInfo, Integer> workerDiskIndexForReplica = new HashMap<>();
-    List<Integer> partitionIdList = new ArrayList<>(partitionIds);
+    List<Integer> partitionIdList = new LinkedList<>(partitionIds);
 
     final int workerSize = workers.size();
     final IntUnaryOperator incrementIndex = v -> (v + 1) % workerSize;