Hello Frédéric Thanks for your response About iops; yes iops drop is good news, but this amount (far more than 50%) was a bit suspicious, and it shouldn't worsen latency. I was worried that slow disk might be doing what fast disk is supposed to do, but that's also not the case, so it looks like io just vanished to near zero. However this very well could be a separate problem/topic. About config defaults; thanks for that link, that's very handy. I checked the diff and tried to point out what might be problematic, these are the configs I tested changing, however did not get any better latency in response: - bluestore_elastic_shared_blob -> false - swapping osd_op_num_shards_hdd and osd_op_num_threads_per_shard that you mentioned - disabling ec partial reads - decreasing rgw thread pool size (I had this set to a higher value than default on reef) - decreasing rgw log trim interval
I haven't tried changing envelope mode, I will, and thanks for the hint, but is it straight forward to upgrade wal back to envelope mode using bluestore tool if downgrade didn't seem to solve the issue? Also meanwhile since my original post I've been doing more tests, and the same case happened on other clusters as well (VMs with no separate fast/slow disks), the only time latency peak didn't happen, was when I tested on a single zone cluster, which is confusing since all my multi zone clusters use separate RGWs for running sync threads, and I'm seeing latency peak on client RGWs, not the sync ones. It might as well be another parameter that's different on the single zone one and I'm missing it, but that's my initial observation. Best regards Moti On Thu, 16 Jul 2026, 19:34 Frédéric Nass via ceph-users, <[email protected]> wrote: > Hi Moti, > > Anything suspicious on 'ceph osd perf' output? > > > NVMe disks' iops went from something around 5-6K constant io per second > to almost zero (less than a hundred per second, on all host/OSDs, with some > non-frequent high peaks), > > That would be pretty good news, actually. This would likely be the result > of the new 'bluefs_wal_envelope_mode' [1]. Its own description says it > "reduces by ~50% the amount of fdatasync syscalls" on the RocksDB WAL. :-) > I don't expect this to explain the performance drop at this time, though. > > I'd recommend that you check the new/old settings and default values > between both versions using our online Config Diff tool here [2]. It might > ring a bell. > > The tool highlights that the default values for 'osd_op_num_shards_hdd' and > 'osd_op_num_threads_per_shard_hdd' were swapped between both versions. I'm > not sure what the impact could be on your cluster in terms of performance, > but if you're using mClock, reverting this change or trying 'wpq' osd > scheduler instead might help. You can run 'ceph config set osd osd_op_queue > wpq' and restart all OSDs to figure out if mClock has anything to do with > what you observe. > > If swapping values or changing the osd scheduler to wpq doesn't help, you > could try disabling 'bluefs_wal_envelope_mode' and running > 'ceph-bluestore-tool --command downgrade-wal-to-v1' (make sure to test > this out on a single OSD first). > > Best regards, > > Frédéric > > [1] > > https://github.com/ceph/ceph/blob/708737d3ec0669a8e01265d9699d6f939debec13/src/common/options/global.yaml.in#L4311 > [2] > > https://analyzer.clyso.com/#/analyzer/config-diff?source=v18.2.4&target=v20.2.1 > > > Frédéric Nass > > Senior Ceph Engineer > > Ceph Ambassador, France > > +49 89 215252-751 <https://call.ctrlq.org/+49%2089%20215252-751> > > [email protected] > > www.clyso.com > > Hohenzollernstr. 27, 80801 Muni > <https://www.google.com/maps/search/Hohenzollernstr.+27,+80801+Muni?entry=gmail&source=g> > ch > > Utting a. A. | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE2754306 > > > > Le jeu. 9 juil. 2026 à 15:33, Moti via ceph-users <[email protected]> a > écrit : > > > Hello everyone! > > I recently upgraded from 18.2.4 to 20.2.1 (so it might be a Squid thing > as > > well as Tentacle) and latency went ~3-4x more on PUT, even worse on GET > and > > HEAD requests. > > > > There is no resource cap or significant increase in usage is visible. > > Neither fast ec nor any other new features are enabled yet, just did the > > upgrade and left it as is. For measuring latency I have a services that > > probes the cluster with consistent load so it's not a difference in load > > either. PG status history does not show any pg operation being heavily in > > process. > > > > Cluster has a hybrid build so OSDs have HDD for data and NVMe for db > block. > > Both ec and replicated pools exist. > > The strange hint was: NVMe disks' iops went from something around 5-6K > > constant io per second to almost zero (less than a hundred per second, on > > all host/OSDs, with some non-frequent high peaks), while the HDD iops is > > not much changed. db blocks on NVMe are still attached to OSDs with no > > issue. > > > > I can't find a clue to explain this. Got the same results on 3 separate > > clusters (all went version 18 --> 20, same hybrid build and configs). > > Nothing in config dump changes (changed defaults) catches my attention to > > have caused this. > > > > Anyone had similar issues? Any tips and ideas on what to check? > > _______________________________________________ > > ceph-users mailing list -- [email protected] > > To unsubscribe send an email to [email protected] > > > _______________________________________________ > ceph-users mailing list -- [email protected] > To unsubscribe send an email to [email protected] > _______________________________________________ ceph-users mailing list -- [email protected] To unsubscribe send an email to [email protected]
