Hmm...We need to fix that as part of configure/Makefile I guess (?).. Since we have done this jemalloc integration originally, we can take that ownership unless anybody sees a problem of enabling tcmalloc/jemalloc with librbd/librados.
<< You have to remove libcmalloc out of your build environment to get this done How do I do that ? I am using Ubuntu and can't afford to remove libc* packages. Thanks & Regards Somnath -----Original Message----- From: Stefan Priebe [mailto:s.pri...@profihost.ag] Sent: Wednesday, August 19, 2015 1:18 PM To: Somnath Roy; Alexandre DERUMIER; Mark Nelson Cc: ceph-devel Subject: Re: Ceph Hackathon: More Memory Allocator Testing Am 19.08.2015 um 22:16 schrieb Somnath Roy: > Alexandre, > I am not able to build librados/librbd by using the following config option. > > ./configure –without-tcmalloc –with-jemalloc Same issue to me. You have to remove libcmalloc out of your build environment to get this done. Stefan > It seems it is building osd/mon/Mds/RGW with jemalloc enabled.. > > root@emsnode10:~/ceph-latest/src# ldd ./ceph-osd > linux-vdso.so.1 => (0x00007ffd0eb43000) > libjemalloc.so.1 => /usr/lib/x86_64-linux-gnu/libjemalloc.so.1 > (0x00007f5f92d70000) > ....... > > root@emsnode10:~/ceph-latest/src/.libs# ldd ./librados.so.2.0.0 > linux-vdso.so.1 => (0x00007ffed46f2000) > libboost_thread.so.1.55.0 => > /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.55.0 (0x00007ff687887000) > liblttng-ust.so.0 => /usr/lib/x86_64-linux-gnu/liblttng-ust.so.0 > (0x00007ff68763d000) > libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007ff687438000) > libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 > (0x00007ff68721a000) > libnss3.so => /usr/lib/x86_64-linux-gnu/libnss3.so > (0x00007ff686ee0000) > libsmime3.so => /usr/lib/x86_64-linux-gnu/libsmime3.so > (0x00007ff686cb3000) > libnspr4.so => /usr/lib/x86_64-linux-gnu/libnspr4.so > (0x00007ff686a76000) > libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 > (0x00007ff686871000) > librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007ff686668000) > libboost_system.so.1.55.0 => > /usr/lib/x86_64-linux-gnu/libboost_system.so.1.55.0 (0x00007ff686464000) > libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 > (0x00007ff686160000) > libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ff685e59000) > libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ff685a94000) > libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 > (0x00007ff68587e000) > liblttng-ust-tracepoint.so.0 => > /usr/lib/x86_64-linux-gnu/liblttng-ust-tracepoint.so.0 (0x00007ff685663000) > liburcu-bp.so.1 => /usr/lib/liburcu-bp.so.1 (0x00007ff68545c000) > liburcu-cds.so.1 => /usr/lib/liburcu-cds.so.1 (0x00007ff685255000) > /lib64/ld-linux-x86-64.so.2 (0x00007ff68a0f6000) > libnssutil3.so => /usr/lib/x86_64-linux-gnu/libnssutil3.so > (0x00007ff685029000) > libplc4.so => /usr/lib/x86_64-linux-gnu/libplc4.so > (0x00007ff684e24000) > libplds4.so => /usr/lib/x86_64-linux-gnu/libplds4.so > (0x00007ff684c20000) > > It is building with libcmalloc always... > > Did you change the ceph makefiles to build librbd/librados with jemalloc ? > > Thanks & Regards > Somnath > > -----Original Message----- > From: ceph-devel-ow...@vger.kernel.org > [mailto:ceph-devel-ow...@vger.kernel.org] On Behalf Of Alexandre > DERUMIER > Sent: Wednesday, August 19, 2015 7:01 AM > To: Mark Nelson > Cc: ceph-devel > Subject: Re: Ceph Hackathon: More Memory Allocator Testing > > Thanks Marc, > > Results are matching exactly what I have seen with tcmalloc 2.1 vs 2.4 vs > jemalloc. > > and indeed tcmalloc, even with bigger cache, seem decrease over time. > > > What is funny, is that I see exactly same behaviour client librbd side, with > qemu and multiple iothreads. > > > Switching both server and client to jemalloc give me best performance on > small read currently. > > > > > > > ----- Mail original ----- > De: "Mark Nelson" <mnel...@redhat.com> > À: "ceph-devel" <ceph-devel@vger.kernel.org> > Envoyé: Mercredi 19 Août 2015 06:45:36 > Objet: Ceph Hackathon: More Memory Allocator Testing > > Hi Everyone, > > One of the goals at the Ceph Hackathon last week was to examine how to > improve Ceph Small IO performance. Jian Zhang presented findings showing a > dramatic improvement in small random IO performance when Ceph is used with > jemalloc. His results build upon Sandisk's original findings that the default > thread cache values are a major bottleneck in TCMalloc 2.1. To further verify > these results, we sat down at the Hackathon and configured the new > performance test cluster that Intel generously donated to the Ceph community > laboratory to run through a variety of tests with different memory allocator > configurations. I've since written the results of those tests up in pdf form > for folks who are interested. > > The results are located here: > > http://nhm.ceph.com/hackathon/Ceph_Hackathon_Memory_Allocator_Testing. > pdf > > I want to be clear that many other folks have done the heavy lifting here. > These results are simply a validation of the many tests that other folks have > already done. Many thanks to Sandisk and others for figuring this out as it's > a pretty big deal! > > Side note: Very little tuning other than swapping the memory allocator and a > couple of quick and dirty ceph tunables were set during these tests. It's > quite possible that higher IOPS will be achieved as we really start digging > into the cluster and learning what the bottlenecks are. > > Thanks, > Mark > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" > in the body of a message to majord...@vger.kernel.org More majordomo > info at http://vger.kernel.org/majordomo-info.html > > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" > in the body of a message to majord...@vger.kernel.org More majordomo > info at http://vger.kernel.org/majordomo-info.html > > ________________________________ > > PLEASE NOTE: The information contained in this electronic mail message is > intended only for the use of the designated recipient(s) named above. If the > reader of this message is not the intended recipient, you are hereby notified > that you have received this message in error and that any review, > dissemination, distribution, or copying of this message is strictly > prohibited. If you have received this communication in error, please notify > the sender by telephone or e-mail (as shown above) immediately and destroy > any and all copies of this message in your possession (whether hard copies or > electronically stored copies). > > N r y b X ǧv ^ ){.n + z ]z {ay ʇڙ ,j f h z w j:+v w j m zZ+ ݢj" !tml= >
