This will be a big win Benno, thanks for driving it! Nice to see that the heap profiling overhead is really low, I'd love to be able to just hit an endpoint on the master or agent and get a memory profile.
I'm a +1 for making it the default, however, I seem to recall hearing that there were some issues with JNI? Ben On Fri, Aug 18, 2017 at 3:49 AM, Benno Evers <bev...@mesosphere.com> wrote: > Hi all, > > I would like to propose bundling jemalloc as a new dependency > under `3rdparty/`, and to link Mesos against this new memory > allocator by default. > > > # Motivation > > The Mesos master and agent binaries are, ideally, very long-running > processes. This makes them susceptible to memory issues, because > even small leaks have a chance to build up over time to the point > where they become problematic. > > We have seen several such issues on our internal Mesos installations, > for example https://issues.apache.org/jira/browse/MESOS-7748 > or https://issues.apache.org/jira/browse/MESOS-7800. > > I imagine any organization running Mesos for an extended period > of time has had its share of similar issues, so I expect this > proposal to be useful for the whole community. > > > # Why jemalloc? > > Given that memory issues tend to be most visible after a given > process has been running for a long time, it would be great to > have the option to enable heap tracking and profiling at runtime, > without having to restart the process. (This ability could then > be connected to a Mesos endpoint, similar to how we can adjust > the log level at runtime) > > The two production-quality memory allocators that have this > ability currently seem to be tcmalloc and jemalloc. Of these, > jemalloc does produce in our experience better and more > detailed statistics. > > > # What is the impact on users who do not need this feature? > > Naturally, not every single user of Mesos will have a need > for this feature. To ensure these users would not experience serious > performance regressions as a result of this change, we conducted > a preliminary set of benchmarks whose results are collected > under https://issues.apache.org/jira/browse/MESOS-7876 > > It turns out that we could probably even expect a small speedup (1% - 5%) > as a nice side-effect of this change. > > Users who compile Mesos themselves would of course have the option > to disable jemalloc at configuration time or replace it with their > memory allocator of choice. > > > > I'm looking forward to hear any thoughts and comments. > > > Thanks, > -- > Benno Evers > Software Engineer, Mesosphere >
