Strongly in favor of this idea and willing to try it once it's available. On Fri, Aug 18, 2017 at 8:37 AM, James Peach <jor...@gmail.com> wrote:
> > > On Aug 18, 2017, at 3:49 AM, Benno Evers <bev...@mesosphere.com> wrote: > > > > Hi all, > > > > I would like to propose bundling jemalloc as a new dependency > > under `3rdparty/`, and to link Mesos against this new memory > > allocator by default. > > I support doing this for all the Mesos executable programs. We have been > running under jemalloc for a couple of years with zero problems and > improved performance (we run a pretty old glibc). > > Note that we must not lib libmesos.so to jemalloc since that is used by > programs that may not be able to tolerate linking in a separate malloc. > > > # Motivation > > > > The Mesos master and agent binaries are, ideally, very long-running > > processes. This makes them susceptible to memory issues, because > > even small leaks have a chance to build up over time to the point > > where they become problematic. > > > > We have seen several such issues on our internal Mesos installations, > > for example https://issues.apache.org/jira/browse/MESOS-7748 > > or https://issues.apache.org/jira/browse/MESOS-7800. > > > > I imagine any organization running Mesos for an extended period > > of time has had its share of similar issues, so I expect this > > proposal to be useful for the whole community. > > > > > > # Why jemalloc? > > > > Given that memory issues tend to be most visible after a given > > process has been running for a long time, it would be great to > > have the option to enable heap tracking and profiling at runtime, > > without having to restart the process. (This ability could then > > be connected to a Mesos endpoint, similar to how we can adjust > > the log level at runtime) > > > > The two production-quality memory allocators that have this > > ability currently seem to be tcmalloc and jemalloc. Of these, > > jemalloc does produce in our experience better and more > > detailed statistics. > > > > > > # What is the impact on users who do not need this feature? > > > > Naturally, not every single user of Mesos will have a need > > for this feature. To ensure these users would not experience serious > > performance regressions as a result of this change, we conducted > > a preliminary set of benchmarks whose results are collected > > under https://issues.apache.org/jira/browse/MESOS-7876 > > > > It turns out that we could probably even expect a small speedup (1% - 5%) > > as a nice side-effect of this change. > > > > Users who compile Mesos themselves would of course have the option > > to disable jemalloc at configuration time or replace it with their > > memory allocator of choice. > > > > > > > > I'm looking forward to hear any thoughts and comments. > > > > > > Thanks, > > -- > > Benno Evers > > Software Engineer, Mesosphere > > -- Cheers, Zhitao Li
