Interesting, I've switched to jemalloc about a month ago while running Hammer. after installing the library and using the /etc/ld.so.preload I am seeing that all ceph-osd processes are indeed using the library. I've upgraded to Jewel a few days ago and see the same picture:
# time lsof |grep ceph-osd |grep jemalloc ceph-osd 1991 ceph mem REG 8,129 207488 525275 /usr/lib/x86_64-linux-gnu/libjemalloc.so.1 ceph-osd 1991 2094 ceph mem REG 8,129 207488 525275 /usr/lib/x86_64-linux-gnu/libjemalloc.so.1 ceph-osd 1991 6689 ceph mem REG 8,129 207488 525275 /usr/lib/x86_64-linux-gnu/libjemalloc.so.1 ceph-osd 2341 ceph mem REG 8,129 207488 525275 /usr/lib/x86_64-linux-gnu/libjemalloc.so.1 ceph-osd 2341 2414 ceph mem REG 8,129 207488 525275 /usr/lib/x86_64-linux-gnu/libjemalloc.so.1 ceph-osd 2341 3813 ceph mem REG 8,129 207488 525275 /usr/lib/x86_64-linux-gnu/libjemalloc.so.1 I've not done anything else apart from apt-get install <library-package> and added it to the ld.so.preload. I've also ran a few dd tests from about 20 vms before and after switching the library and see a significant performance increase for parallel writes. The increase was around 5x, which I couldn't believe at first, but repeated tests verified that. To me it looks like the libbrary is being used, but please advise if it is otherwise. Thanks Andrei ----- Original Message ----- > From: "Joshua M. Boniface" <[email protected]> > To: "ceph-users" <[email protected]> > Sent: Tuesday, 24 May, 2016 19:00:55 > Subject: Re: [ceph-users] using jemalloc in trusty > Hello list: > > I've been building my own Ceph Debian Jessie packages (and QEMU too) to get > jemalloc support, and in Infernalis I was explicitly setting a dependency in > the control file which seemed to work. However, that option is gone in Jewel, > replaced with this /etc/default/ceph preload. Without any code changes from > default, and enabling the jemalloc line in that file, I'm seeing both > libraries > in use by every daemon, e.g.: > >> ms_pipe_r 8659 31756 ceph mem REG 253,0 640600 >> 21056 /usr/lib/libtcmalloc.so.4.2.2 >> ms_pipe_r 8659 31756 ceph mem REG 253,0 223936 >> 6829 /usr/lib/x86_64-linux-gnu/libjemalloc.so.1 >> ms_pipe_w 8659 31757 ceph mem REG 253,0 640600 >> 21056 /usr/lib/libtcmalloc.so.4.2.2 >> ms_pipe_w 8659 31757 ceph mem REG 253,0 223936 >> 6829 /usr/lib/x86_64-linux-gnu/libjemalloc.so.1 > > Given the messages in this thread, it seems that the jemalloc library isn't > actually being used? But if so, why would it be loaded (and why would tcmalloc > *also* be loaded)? And if we still need to add explicit support, does anyone > have any advice for how to achieve this? > > My build procedure is quite simply: > > $ git clone https://github.com/ceph/ceph.git && cd ceph > $ sudo dpkg-buildpackage -us -uc -j8 > > Followed by adding the generated packages to my local repo and standard > "apt-get > install". And yes, the main Ceph packages have dependencies on "libtcmalloc-*" > (just tested trying to remove them). > > -- > Joshua M. Boniface > Linux System Ærchitect > Sigmentation fault. Core dumped. > > On 24/05/16 01:49 PM, Alexandre DERUMIER wrote: >>>> Is it true for Xenial too or only for Trusty? I don't want to rebuild >>>> Jewel on >>>> xenial hosts... >> yes, for xenial (and debian wheezy/jessie too). >> >> I don't don't why they are LD_PRELOAD commented in /etc/default/ceph, >> because it's really don't do nothing, if tcmalloc is present. >> >> you could try to remove tcmalloc packages (libgoogle-perftools4), >> >> but I think that ceph packages have dependencies on it .... >> >> >> >> ----- Mail original ----- >> De: "Max A. Krasilnikov" <[email protected]> >> Cc: "ceph-users" <[email protected]> >> Envoyé: Mardi 24 Mai 2016 19:33:48 >> Objet: Re: [ceph-users] using jemalloc in trusty >> >> Hello! >> >> On Mon, May 23, 2016 at 02:34:37PM +0000, Somnath.Roy wrote: >> >>> You need to build ceph code base to use jemalloc for OSDs..LD_PRELOAD won't >>> work.. >> Is it true for Xenial too or only for Trusty? I don't want to rebuild Jewel >> on >> xenial hosts... >> > > _______________________________________________ > ceph-users mailing list > [email protected] > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
