Hi! We also observe the same behavior on our test Hammer install, and I wrote about it some time ago:
<http://permalink.gmane.org/gmane.comp.file-systems.ceph.user/22609>http://permalink.gmane.org/gmane.comp.file-systems.ceph.user/22609 Jan Schremes give us some suggestions in thread, but we still not got any positive results - TCMalloc usage is high. The usage is lowered to <10%, when disable crc in messages, disable debug and disable cephx auth, but this is od course not for production use. Also we got a different trace, while performin FIO-RBD benchmarks on ssd pool: --- 46,07% [kernel] [k] _raw_spin_lock 6,51% [kernel] [k] mb_cache_entry_alloc 5,74% libtcmalloc.so.4.2.2 [.] tcmalloc::CentralFreeList::FetchFromOneSpans(int, void**, void**) 5,50% libtcmalloc.so.4.2.2 [.] tcmalloc::SLL_Next(void*) 3,86% libtcmalloc.so.4.2.2 [.] TCMalloc_PageMap3<35>::get(unsigned long) const 2,73% libtcmalloc.so.4.2.2 [.] tcmalloc::CentralFreeList::ReleaseToSpans(void*) 0,69% libtcmalloc.so.4.2.2 [.] tcmalloc::CentralFreeList::ReleaseListToSpans(void*) 0,69% libtcmalloc.so.4.2.2 [.] tcmalloc::PageHeap::GetDescriptor(unsigned long) const 0,64% libtcmalloc.so.4.2.2 [.] tcmalloc::SLL_PopRange(void**, int, void**, void**) --- I dont clearly understand, what's happening in this case: ssd pool is connected to the same host, but different controller (C60X onboard instead of LSI2208), io scheduler set to noop, pool is gathered from 4х400Gb Intel DC S3700 and have to perform better, I think - more than 30-40 kops. But we got the trace above and no more then 12-15 kiops. Where can be a problem? Megov Igor CIO, Yuterra ________________________________ От: ceph-users <ceph-users-boun...@lists.ceph.com> от имени YeYin <ey...@qq.com> Отправлено: 17 августа 2015 г. 12:58 Кому: ceph-users Тема: [ceph-users] tcmalloc use a lot of CPU Hi, all, When I do performance test with rados bench, I found tcmalloc consumed a lot of CPU: Samples: 265K of event 'cycles', Event count (approx.): 104385445900 + 27.58% libtcmalloc.so.4.1.0 [.] tcmalloc::CentralFreeList::FetchFromSpans() + 15.25% libtcmalloc.so.4.1.0 [.] tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, + 12.20% libtcmalloc.so.4.1.0 [.] tcmalloc::CentralFreeList::ReleaseToSpans(void*) + 1.63% perf [.] append_chain + 1.39% libtcmalloc.so.4.1.0 [.] tcmalloc::CentralFreeList::ReleaseListToSpans(void*) + 1.02% libtcmalloc.so.4.1.0 [.] tcmalloc::CentralFreeList::RemoveRange(void**, void**, int) + 0.85% libtcmalloc.so.4.1.0 [.] 0x0000000000017e6f + 0.75% libtcmalloc.so.4.1.0 [.] tcmalloc::ThreadCache::IncreaseCacheLimitLocked() + 0.67% libc-2.12.so [.] memcpy + 0.53% libtcmalloc.so.4.1.0 [.] operator delete(void*) Ceph version: # ceph --version ceph version 0.87.2 (87a7cec9ab11c677de2ab23a7668a77d2f5b955e) Kernel version: 3.10.83 Is this phenomenon normal?Is there any idea about this problem? Thanks. Ye
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
