Hi Magnus,
Here is the brief overview for the relevant perf counters:
- 'issued deferred writes' denotes amount of user write ops which have
got deferred processing and have been persisted at Key-Value DB (The
first stage of deferred write procedure).
- 'submitted deferred writes' denotes amount of writes which have been
submitted to main device to reach their final disk location. (the second
stage of deferred write procedure).
Additionally there is the last stage which is staged deferred writes
cleanup from KV. But there is no relevant counter to track this state
though at the moment.
Frankly speaking I recall no modifications to
bluestore_prefer_deferred_size_hdd happened to Squid. The only
not-that-recent PR which might look relevant is
https://github.com/ceph/ceph/pull/48490 but it's been backported back to
Pacific and Quincy. May be you had had another upgrade(s) prior to R->S
one recently which triggered the regression?
It would be great to get more information on the issue including perf
counter dumps and more details on that performance degradation and write
patters which are affected.
Thanks,
Igor
On 4/14/2026 5:00 PM, Magnus Grönlund via ceph-users wrote:
Hi All,
We are running a 5 year old 15 node Ceph cluster with spinners for data and
NVMe for DB/Wal.
After the recent upgrade to Squid (from Reef) some users reported issues
with fluctuating (often bad) disk performance especially affecting k8s etcd.
They point to the change to bluestore_prefer_deferred_size_hdd claiming
that it's detrimental to the "synchronous write patter" of escd.
They also point to submitted_deferred_writes and issued_deferred_writes
claiming that there is a massive and growing backlog of deferred writes.
I'm not 100% correct that this is the issue but I'm having a hard time
finding any relevant information (bad google-fu?) especially about what
submitted and issued deferred writes actually represents and why the
difference between issued - submitted keeps growing.
Anyone got some insights to share?
Thanks in advance
Magnus
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
