This is an automated email from the ASF dual-hosted git repository.
ethanfeng pushed a commit to branch branch-0.5
in repository https://gitbox.apache.org/repos/asf/celeborn.git
The following commit(s) were added to refs/heads/branch-0.5 by this push:
new 4eea40e1e [CELEBORN-1403] Update celeborn configuration about offheap
memory
4eea40e1e is described below
commit 4eea40e1ee82c9e9a394c40c7142ca9edf17030b
Author: mingji <[email protected]>
AuthorDate: Tue Jun 11 15:44:25 2024 +0800
[CELEBORN-1403] Update celeborn configuration about offheap memory
### What changes were proposed in this pull request?
Update configuration guide.
### Why are the changes needed?
Remove outdated informations about worker's offheap memory.
### Does this PR introduce _any_ user-facing change?
NO.
### How was this patch tested?
No need to test.
Closes #2552 from FMX/b1403.
Authored-by: mingji <[email protected]>
Signed-off-by: mingji <[email protected]>
(cherry picked from commit 22ecbbd157c972141080f67f0bec12527c4537a0)
Signed-off-by: mingji <[email protected]>
---
docs/configuration/index.md | 11 +++++++----
1 file changed, 7 insertions(+), 4 deletions(-)
diff --git a/docs/configuration/index.md b/docs/configuration/index.md
index 9d620a0b5..c657f32b5 100644
--- a/docs/configuration/index.md
+++ b/docs/configuration/index.md
@@ -36,14 +36,17 @@ Off-heap memory requirement can be estimated as below:
```
numDirs = `celeborn.worker.storage.dirs` # the amount of directory
will be used by Celeborn storage
bufferSize = `celeborn.worker.flusher.buffer.size` # the amount of memory
will be used by a single flush buffer
-off-heap-memory = bufferSize * estimatedTasks * 2 + network memory
+off-heap-memory = (disk buffer * disks) + network memory # the disk
buffer is a logical memory region that stores shuffle data received from
network
+ # shuffle data
will be flushed to disks through write tasks
+ # the amount of
disk buffer can be set to 1GB or larger for each disk according to the
difference of your disk speed and network speed
```
-For example, if a Celeborn worker has 10 storage directories or disks and the
buffer size is set to 256 KiB.
-The necessary off-heap memory is 10 GiB.
+For example, if a Celeborn worker give each disk 1GiB memory and the buffer
size is set to 256 KB.
+Celeborn worker can support up to 4096 concurrent write tasks for each disk.
+If this worker has 10 disks, the offheap memory should be set to 12GB.
Network memory will be consumed when netty reads from a TCP channel, there
will need some extra
-memory. Empirically, Celeborn worker off-heap memory should be set to
`(numDirs * bufferSize * 1.2)`.
+memory. Empirically, Celeborn worker off-heap memory should be set to `((disk
buffer * disks) * 1.2)`.
## All Configurations
