Hi Moti, Anything suspicious on 'ceph osd perf' output?
> NVMe disks' iops went from something around 5-6K constant io per second to almost zero (less than a hundred per second, on all host/OSDs, with some non-frequent high peaks), That would be pretty good news, actually. This would likely be the result of the new 'bluefs_wal_envelope_mode' [1]. Its own description says it "reduces by ~50% the amount of fdatasync syscalls" on the RocksDB WAL. :-) I don't expect this to explain the performance drop at this time, though. I'd recommend that you check the new/old settings and default values between both versions using our online Config Diff tool here [2]. It might ring a bell. The tool highlights that the default values for 'osd_op_num_shards_hdd' and 'osd_op_num_threads_per_shard_hdd' were swapped between both versions. I'm not sure what the impact could be on your cluster in terms of performance, but if you're using mClock, reverting this change or trying 'wpq' osd scheduler instead might help. You can run 'ceph config set osd osd_op_queue wpq' and restart all OSDs to figure out if mClock has anything to do with what you observe. If swapping values or changing the osd scheduler to wpq doesn't help, you could try disabling 'bluefs_wal_envelope_mode' and running 'ceph-bluestore-tool --command downgrade-wal-to-v1' (make sure to test this out on a single OSD first). Best regards, Frédéric [1] https://github.com/ceph/ceph/blob/708737d3ec0669a8e01265d9699d6f939debec13/src/common/options/global.yaml.in#L4311 [2] https://analyzer.clyso.com/#/analyzer/config-diff?source=v18.2.4&target=v20.2.1 Frédéric Nass Senior Ceph Engineer Ceph Ambassador, France +49 89 215252-751 <https://call.ctrlq.org/+49%2089%20215252-751> [email protected] www.clyso.com Hohenzollernstr. 27, 80801 Munich Utting a. A. | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE2754306 Le jeu. 9 juil. 2026 à 15:33, Moti via ceph-users <[email protected]> a écrit : > Hello everyone! > I recently upgraded from 18.2.4 to 20.2.1 (so it might be a Squid thing as > well as Tentacle) and latency went ~3-4x more on PUT, even worse on GET and > HEAD requests. > > There is no resource cap or significant increase in usage is visible. > Neither fast ec nor any other new features are enabled yet, just did the > upgrade and left it as is. For measuring latency I have a services that > probes the cluster with consistent load so it's not a difference in load > either. PG status history does not show any pg operation being heavily in > process. > > Cluster has a hybrid build so OSDs have HDD for data and NVMe for db block. > Both ec and replicated pools exist. > The strange hint was: NVMe disks' iops went from something around 5-6K > constant io per second to almost zero (less than a hundred per second, on > all host/OSDs, with some non-frequent high peaks), while the HDD iops is > not much changed. db blocks on NVMe are still attached to OSDs with no > issue. > > I can't find a clue to explain this. Got the same results on 3 separate > clusters (all went version 18 --> 20, same hybrid build and configs). > Nothing in config dump changes (changed defaults) catches my attention to > have caused this. > > Anyone had similar issues? Any tips and ideas on what to check? > _______________________________________________ > ceph-users mailing list -- [email protected] > To unsubscribe send an email to [email protected] > _______________________________________________ ceph-users mailing list -- [email protected] To unsubscribe send an email to [email protected]
