a7i opened a new issue, #3867:
URL: https://github.com/apache/celeborn/issues/3867
Hello Celeborn Community!
As the title suggests, we are having disk performance issues. I will take
the first few sections to reveal our setup. I'm hoping to see what other are
doing to remediate some of these bottlebecks.
### Deployment
Running Celeborn **0.7.0** with Spark.
- Workers pinned per AZ (`celeborn.tags` + client `tagsExpr`)
- Push replication off (single copy in the AZ)
- One worker pod per shuffle node
- Shuffle data on **instance store** (not EBS)
- Single `celeborn.worker.storage.dirs` entry (`/mnt/disk1`)
### Disk layout (shuffle nodes)
Separate boot volume; data NVMe disks go into one LVM striped volume group.
Celeborn only sees the kubelet mount.
*i8ge.6xlarge: 2× data NVMe. i8ge.12xlarge: 4× data NVMe (same layout, wider
stripe).*
```mermaid
flowchart TB
boot["Boot NVMe — root FS only"]
subgraph data["Data NVMe ×2 or ×4"]
d1["nvme0n1"]
d2["nvme1n1"]
dMore["…"]
end
lvm["LVM VG nvme — striped 256K"]
lvPods["LV nvme-kubelet"]
lvCtr["LV nvme-containerd"]
dm["dm-0"]
ext4["ext4 noatime"]
kubelet["/var/lib/kubelet/pods"]
empty["Pod emptyDir"]
worker["Celeborn /mnt/disk1"]
boot ~~~ data
d1 & d2 & dMore --> lvm
lvm --> lvPods
lvm --> lvCtr
lvPods --> dm --> ext4 --> kubelet --> empty --> worker
```
```
boot NVMe data NVMe (×2 or ×4)
| |
/ (root) LVM VG "nvme" (stripe 256K)
|
+-----------------+------------------+
| |
nvme-kubelet nvme-containerd
|
dm-0
|
ext4
|
/var/lib/kubelet/pods (per-pod UID dirs)
|
emptyDir (sizeLimit)
|
Celeborn /mnt/disk1
```
Under prod bursts, **`dm-0` IO util** pegs (~100%) and that’s the limiter.
NIC and master CPU aren’t the story. Trying to figure out if we’re missing a
Celeborn knob or if it’s more workers / less shuffle data.
### What we’ve already tried (worker / node)
On **i8ge.6xlarge** (2× ~7.5TB NVMe, 24 vCPU, 192 GiB):
- Congestion watermarks tuned so we don’t exclude the whole AZ during
absorbable Netty bursts (disk still healthy, lots of free space)
- SSD flusher threads **16 → 32 → 64** (64 helped flush-queue depth and job
wall time in sustained tests; 128 was worse)
- Flusher buffer **512K → 2M** (helped queue depth under the same load)
- Node readahead turned up: no measurable gain on fetch-like reads (8m
chunks, qd1)
- fio on the **same path Celeborn uses** (scratch under a live pod UID under
`/var/lib/kubelet/pods`, `direct=1`, 512K blocks)
| Pattern | fio ceiling | Celeborn under load |
| --- | --- | --- |
| Sequential write | 3.08 GB/s (~92% of ceiling) | 2.83 GB/s |
| Sequential read | 6.23 GB/s (~70% of ceiling) | 4.34 GB/s |
| Mixed 70/30 read / write | 6.24 / 2.68 GB/s (~81% of ceiling) | 4.34 /
2.83 GB/s |
Both data NVMe devices at ~100% in fio and in the load test (via `dm-0`
slaves). More aggressive fio only buys a few percent on writes. Readahead and
AMI queue tuning didn’t move the needle.
Rolling **i8ge.12xlarge** (4× NVMe in one striped LV, same Celeborn layout)
and **more workers per AZ**. Dashboard still screams disk; 12xlarge helps a bit
on peak write MB/s but not a 2× step. Still finishing fio on 12xlarge.
### Prod-ish worker settings
- `celeborn.worker.flusher.ssd.threads=64`
- `celeborn.worker.flusher.buffer.size=2M`
- Congestion control enabled (watermarks derived from direct-memory cap)
- Large `MaxDirectMemorySize` (120g on 6xlarge, 240g on 12xlarge)
- `celeborn.client.*` does **not** come from our server ConfigMap; Spark
uses defaults unless the app repo sets something
### What I’m asking
Not a bug report (workers healthy, flush latency fine, tons of disk free).
When you’re already within ~10% of fio on writes on current-gen AWS NVMe, what
do you recommend?
1. Fair to treat **per-worker disk bandwidth** as a hard cap and optimize
around it (more workers, less shuffle bytes) instead of more flusher/congestion
tuning?
2. **Client / Spark**: compression, partition count, in-flight limits
(`maxReqsInFlight`, etc.)? Anything that works when the worker is
drain-limited, not Netty-limited?
3. **Read-heavy** phases: guidance on **direct memory vs page cache** on
big-memory boxes? Sustained test had reads as a large share of disk traffic;
fio read ceiling was higher than Celeborn hit.
4. **Congestion / exclusion**: all workers in an AZ excluded for “high
workload” while disks still look fine. Pattern to avoid registration failures
when the limiter is transient memory pressure, not disk health?
5. **Sizing**: bigger box with more NVMe vs more workers per AZ. Any rule of
thumb on cloud instance store?
Happy to add 12xlarge fio when it’s done. If the answer is mostly “put this
in the tuning guide,” I’ll do a doc PR once I know what you’d want said.
Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
