[ceph-users] Re: Low number of inflight compare to op_wip

Tue, 16 Dec 2025 05:37:32 -0800 


> I'm tryting to debug low performance on nvme-based cluster.

Which specific drives?

> I have 24 NVME in 4 servers, plenty of CPU, cluster perfectly balanced, no 
> scrubbing or replication atm, 1024 pg for 24 OSD/17TB.
> 
> I expect to see some performance (~250k IOPS total, 50% r/w, few hundred 
> volumes with capped by IOPS, pre-warmed). I see about half of it.
> 
> I looked at drive utilization, it's about 70% (per atop)

That's meaningless for SSDs.


> but I've noticed, that in-flight for drives is, basically, around 1. That 
> means, that at a given time only one request is processed. This is match 
> match for OSD count/3 /latency formula, and with one in-flight nvme is 
> showing about 10% of it's specs (Intel, DC grade).

Have you updated to the latest firmware with SST?

> 
> I looked at osd (they loaded uniformely, so any of them shows the same 
> results).
> 
> I see   "op_wip": 10-27, but in-flight value is about 0-2, mostly around 1.
> 
> I can't get away from the feeling that somehow osd is doing operations 
> (almost) sequential. I already played with osd_op_num_threads_per_shard (4), 
> osd_op_num_shards (8), set mclock profile to high_client_ops, 
> ms_async_op_threads 24, and it can't get more inflight ios.
> 
> I feel I miss something. How to make Ceph to send more requests to 
> underlaying NVME?
> 
> _______________________________________________
> ceph-users mailing list -- [email protected]
> To unsubscribe send an email to [email protected]
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]






















					Reply via email to