Re: [ceph-users] High Load and High Apply Latency

Hi John, I am trying to squize extra performance from my test cluster too Dell R 620 with PERC 710 , RAID0, 10 GB network Would you be willing to share your controller and kernel configuration ? For example, I am using BIOS profile 'Performance" with the following added to /etc/default/kernel

Re: [ceph-users] High Load and High Apply Latency

Petrini [mailto:jpetr...@coredial.com] Sent: zaterdag 17 februari 2018 1:06 To: David Turner Cc: ceph-users Subject: Re: [ceph-users] High Load and High Apply Latency I thought I'd follow up on this just in case anyone else experiences similar issues. We ended up increasing the tcmalloc thread cache

Re: [ceph-users] High Load and High Apply Latency

I thought I'd follow up on this just in case anyone else experiences similar issues. We ended up increasing the tcmalloc thread cache size and saw a huge improvement in latency. This got us out of the woods because we were finally in a state where performance was good enough that it was no longer

Re: [ceph-users] High Load and High Apply Latency

Hello, Looking at perf top it looks as though Ceph is spending most of it's CPU cycles on tcmalloc. Looking around online i found that this is a known issue and in fact I found this guide on how to increase the tcmalloc thread cache size:

Re: [ceph-users] High Load and High Apply Latency

Another strange thing I'm seeing is that two of the nodes in the cluster have some OSD's with almost no activity. If I watch top long enough I'll eventually see cpu utilization on these osds but for the most part they sit a 0% cpu utilization. I'm not sure if this is expected behavior or not

Re: [ceph-users] High Load and High Apply Latency

Hi David, Thanks for the info. The controller in the server (perc h730) was just replaced and the battery is at full health. Prior to replacing the controller I was seeing very high iowait when running iostat but I no longer see that behavior - just apply latency when running ceph osd perf. Since

Re: [ceph-users] High Load and High Apply Latency

We show high disk latencies on a node when the controller's cache battery dies. This is assuming that you're using a controller with cache enabled for your disks. In any case, I would look at the hardware on the server. On Thu, Dec 14, 2017 at 10:15 AM John Petrini

Re: [ceph-users] High Load and High Apply Latency

Anyone have any ideas on this? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] High Load and High Apply Latency

Hi List, I've got a 5 OSD node cluster running hammer. All of the OSD servers are identical but one has about 3-4x higher load than the others and the OSD's in this node are reporting high apply latency. The cause of the load appears to be the OSD processes. About half of the OSD processes are