xleoken commented on PR #2119:
URL:
https://github.com/apache/incubator-celeborn/pull/2119#issuecomment-1837976661
hi @FMX @onebox-li @RexXiong take a look again when free, thanks.
Let me make a summary, this patch aims to improve the readability, the patch
had three version, and I used the V3 finally.
If so, we can use the same algorithm when offering slots and offering extra
slots in `Master#handleRequestSlots`.
But @mridulm still believes that there is no need to change, despite his
objections, I am very grateful to him for sharing his thoughts.
**performance**
As @mridulm, have tested, 32.588ns/op vs 7.152ns/op(based on LinkedList),
seems that it can't affect the handleRequestSlots performance.
e.g (based on LinkedList)
1 millions of shuffles, the delta is
100, 0000 * (32ns - 7ns) = 25000000 ns < 1s
**V1, concat two list**
```
A = List(1, 3, 5, 6, 7, 8)
B = A + A = (1, 3, 5, 6, 7, 8, 1, 3, 5, 6, 7 ,8)
let start = 3, end = 7
C = (6, 7, 8, 1)
```
**V2, use startIndex, endIndex**
```
val startIndex = Random.nextInt(numAvailableWorkers)
val endIndex = startIndex + numWorkers - 1
val selectedWorkers = new util.ArrayList[WorkerInfo](numWorkers)
for (index <- startIndex to endIndex) {
val realIndex = index % numAvailableWorkers
selectedWorkers.add(availableWorkers.get(realIndex))
}
```
**V3, use the same algorithm as offering extra slots does.**
```
val selectedWorkers = new util.ArrayList[WorkerInfo](numWorkers)
var index = Random.nextInt(numAvailableWorkers)
(1 to numWorkers).foreach(_ => {
selectedWorkers.add(availableWorkers.get(index))
index = (index + 1) % numAvailableWorkers
})
```
**_Refer_**
https://github.com/apache/incubator-celeborn/blob/1c7cd1bd1392c8d3fd4396c539696dc79835f8dc/master/src/main/scala/org/apache/celeborn/service/deploy/master/Master.scala#L738-L750
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
