Hi John,
I am trying to squize extra performance from my test cluster too
Dell R 620 with PERC 710 , RAID0, 10 GB network
Would you be willing to share your controller and kernel configuration ?
For example, I am using BIOS profile 'Performance" with the following
added to /etc/default/kernel
i
Petrini [mailto:jpetr...@coredial.com]
Sent: zaterdag 17 februari 2018 1:06
To: David Turner
Cc: ceph-users
Subject: Re: [ceph-users] High Load and High Apply Latency
I thought I'd follow up on this just in case anyone else experiences
similar issues. We ended up increasing the tcmalloc thread
I thought I'd follow up on this just in case anyone else experiences
similar issues. We ended up increasing the tcmalloc thread cache size and
saw a huge improvement in latency. This got us out of the woods because we
were finally in a state where performance was good enough that it was no
longer i
Hello,
Looking at perf top it looks as though Ceph is spending most of it's CPU
cycles on tcmalloc. Looking around online i found that this is a known
issue and in fact I found this guide on how to increase the tcmalloc thread
cache size:
https://swamireddy.wordpress.com/2017/01/27/increase-tcmall
Another strange thing I'm seeing is that two of the nodes in the cluster
have some OSD's with almost no activity. If I watch top long enough I'll
eventually see cpu utilization on these osds but for the most part they sit
a 0% cpu utilization. I'm not sure if this is expected behavior or not
though
Hi David,
Thanks for the info. The controller in the server (perc h730) was just
replaced and the battery is at full health. Prior to replacing the
controller I was seeing very high iowait when running iostat but I no
longer see that behavior - just apply latency when running ceph osd perf.
Since
We show high disk latencies on a node when the controller's cache battery
dies. This is assuming that you're using a controller with cache enabled
for your disks. In any case, I would look at the hardware on the server.
On Thu, Dec 14, 2017 at 10:15 AM John Petrini wrote:
> Anyone have any ide
Anyone have any ideas on this?
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Hi List,
I've got a 5 OSD node cluster running hammer. All of the OSD servers are
identical but one has about 3-4x higher load than the others and the OSD's
in this node are reporting high apply latency.
The cause of the load appears to be the OSD processes. About half of the
OSD processes are us