> On Aug 18, 2017, at 3:49 AM, Benno Evers <bev...@mesosphere.com> wrote: > > Hi all, > > I would like to propose bundling jemalloc as a new dependency > under `3rdparty/`, and to link Mesos against this new memory > allocator by default.
I support doing this for all the Mesos executable programs. We have been running under jemalloc for a couple of years with zero problems and improved performance (we run a pretty old glibc). Note that we must not lib libmesos.so to jemalloc since that is used by programs that may not be able to tolerate linking in a separate malloc. > # Motivation > > The Mesos master and agent binaries are, ideally, very long-running > processes. This makes them susceptible to memory issues, because > even small leaks have a chance to build up over time to the point > where they become problematic. > > We have seen several such issues on our internal Mesos installations, > for example https://issues.apache.org/jira/browse/MESOS-7748 > or https://issues.apache.org/jira/browse/MESOS-7800. > > I imagine any organization running Mesos for an extended period > of time has had its share of similar issues, so I expect this > proposal to be useful for the whole community. > > > # Why jemalloc? > > Given that memory issues tend to be most visible after a given > process has been running for a long time, it would be great to > have the option to enable heap tracking and profiling at runtime, > without having to restart the process. (This ability could then > be connected to a Mesos endpoint, similar to how we can adjust > the log level at runtime) > > The two production-quality memory allocators that have this > ability currently seem to be tcmalloc and jemalloc. Of these, > jemalloc does produce in our experience better and more > detailed statistics. > > > # What is the impact on users who do not need this feature? > > Naturally, not every single user of Mesos will have a need > for this feature. To ensure these users would not experience serious > performance regressions as a result of this change, we conducted > a preliminary set of benchmarks whose results are collected > under https://issues.apache.org/jira/browse/MESOS-7876 > > It turns out that we could probably even expect a small speedup (1% - 5%) > as a nice side-effect of this change. > > Users who compile Mesos themselves would of course have the option > to disable jemalloc at configuration time or replace it with their > memory allocator of choice. > > > > I'm looking forward to hear any thoughts and comments. > > > Thanks, > -- > Benno Evers > Software Engineer, Mesosphere
