Re: [Proposal] Use jemalloc as default memory allocator for Mesos

[andrew]

I looked at the Windows build a bit closer, and the Cygwin requirement 
appears to simply be for the use of autogen to generate headers. Once 
the headers are generated, the rest of the build is driven by a 
committed Visual Studio 2015 solution, which means that jemalloc can be 
built natively on Windows using MSVC.

It should be possibly to replace autogen with some other tooling in 
order to generate the required headers. However, this is work we would 
likely have to do and of course submit upstream.

Furthermore, the committed solution itself is a bad idea, as it won't 
"just work" for Visual Studio 2017. We would want to instead write a 
CMake build for jemalloc to replace the committed solution upstream, so 
that VS 2015, 2017, etc. solutions can just be generated. This is what 
we do for Mesos, and have done for other dependencies such as ZooKeeper.

So in summary: it looks like jemalloc is usable on Windows, but it will 
require further effort to integrate into the build cleanly.

On 08/30/2017 6:26 pm, Till Toenshoff wrote:
It appears that jemalloc does support Windows (64bit)
See: https://github.com/jemalloc/jemalloc/blob/dev/msvc/ReadMe.txt
<https://github.com/jemalloc/jemalloc/blob/dev/msvc/ReadMe.txt>

tcmalloc on the other hand appears to only offer a minimal variant on 
Windows.
See: https://github.com/gperftools/gperftools
<https://github.com/gperftools/gperftools> - grep for "COMPILING ON
NON-LINUX SYSTEMS”
See: https://github.com/gperftools/gperftools/blob/master/INSTALL
<https://github.com/gperftools/gperftools/blob/master/INSTALL> - grep
for “Windows  (MSVC, Cygwin, and MinGW)"

So both options rely on Cygwin or MinGW for building - a requirement
that the Mesos build itself does not have and proves your point of
stuff not really “just working” at least when it comes to the build
step of those packages.

Seems that without trying it, we won’t find out if jemalloc works as
hoped on Windows for us - the Firefox project results however are
encouraging. On the other hand, if it doesn’t work, we could simply
decide to disable it on Windows just like some other Mesos features
will remain disabled on that platform unless someone decides to port
them -  e.g. SASL based authn.

On Aug 25, 2017, at 3:28 PM, Benno Evers <bev...@mesosphere.com> 
wrote:

Hi Jeff,

from looking around on the internet, it seems like Firefox builds with
jemalloc on all platforms, and I've also seen reports of people
successfully using tcmalloc heap profiling on windows. I'm afraid I 
don't
currently have a Windows machine with development environment set up, 
so I
can't provide direct user experience.

In the worst case, we would have to disable jemalloc-support on 
windows,
but I hope it won't be necessary.

Since you probably have more experience with memory management on 
windows,
is there a reason to suspect that it should or shouldn't work?

Best regards,
Benno

On Wed, Aug 23, 2017 at 6:16 PM, Jeff Coffler <
jeff.coff...@microsoft.com.invalid> wrote:

Hi Benno,

What's the availability of both jemalloc and tcmalloc on the Windows
platform? Do the products work there properly?

There are solutions that I know work on Windows (from past work I've
done). I'm unsure about either jemalloc and tcmalloc, however.

Thanks,

/Jeff

-----Original Message-----
From: Benno Evers [mailto:bev...@mesosphere.com]
Sent: Tuesday, August 22, 2017 3:16 AM
To: dev@mesos.apache.org
Subject: Re: [Proposal] Use jemalloc as default memory allocator for 
Mesos

Hi Alexander,

in general, jemalloc and tcmalloc are very similar, and seem to be 
taking
ideas from each other (in fact the jeprof executable started as a 
copy of
pprof and there are still references the pprof documentation in some
comments)

From what I've seen, the main difference is that the profiling seems
better-suited to multi-threaded programs, in particular the profile 
file
format includes per-thread memory statistics and the profiling 
features can
be turned on and off individually per thread. From an API 
perspective, all
settings can be accessed by the mallctl() function, while it seems 
that
tcmalloc requires some options to be set by environment variable (
https://na01.safelinks.protection.outlook.com/?url=
https%3A%2F%2Fgperftools.github.io%2Fgperftools%
2Fheapprofile.html&data=02%7C01%7CJeff.Coffler%40microsoft.com%
7Ccb0bfb1eb3e242c0dd4108d4e946d709%7C72f988bf86f141af91ab2d7cd011
db47%7C1%7C0%7C636389937852256730&sdata=IQeb2%
2BpcrWRQ8yvdTgOEHfyplgC36dy73nnXswdPamo%3D&reserved=0). Finally, I 
also
found the documentation to be more thorough.

But again, the two are very similar, so I think the main decision 
here
isn't whether to choose jemalloc or tcmalloc but whether to switch to 
a
custom memory allocator that has support for profiling heap memory 
usage.


On Mon, Aug 21, 2017 at 4:26 PM, Alexander Rojas 
<alexan...@mesosphere.io>
wrote:

Hi Benno,

This does sound like a great addition to Mesos. Can you however
explain how jemalloc is better than tcmalloc? I think that for such
important change, we probably need some more information.

Your comment in MESOS-7876 mentions that we already have tcmalloc
since it is part of gperftools, so I would like to have a whole
picture of the advantages and disadvantages of both options.

Alexander Rojas
alexan...@mesosphere.io




On 18. Aug 2017, at 12:49, Benno Evers <bev...@mesosphere.com> 
wrote:

Hi all,

I would like to propose bundling jemalloc as a new dependency under
`3rdparty/`, and to link Mesos against this new memory allocator by
default.


# Motivation

The Mesos master and agent binaries are, ideally, very long-running
processes. This makes them susceptible to memory issues, because
even small leaks have a chance to build up over time to the point
where they become problematic.

We have seen several such issues on our internal Mesos
installations, for example
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fiss
ues.apache.org%2Fjira%2Fbrowse%2FMESOS-7748&data=02%7C01%7CJeff.Coff
ler%40microsoft.com%7Ccb0bfb1eb3e242c0dd4108d4e946d709%7C72f988bf86f
141af91ab2d7cd011db47%7C1%7C0%7C636389937852266742&sdata=L016YGyEkK5
0WtvhgSNS%2FT5ntkkd9qINorRI2Utp5lk%3D&reserved=0
or https://na01.safelinks.protection.outlook.com/?url=
https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FMESOS-
7800&data=02%7C01%7CJeff.Coffler%40microsoft.com%
7Ccb0bfb1eb3e242c0dd4108d4e946d709%7C72f988bf86f141af91ab2d7cd011
db47%7C1%7C0%7C636389937852266742&sdata=IrzDO6o1VL9a8eGJIW3jKbWXk6U4fH
Fn3Xbn4po1r3c%3D&reserved=0.

I imagine any organization running Mesos for an extended period of
time has had its share of similar issues, so I expect this proposal
to be useful for the whole community.


# Why jemalloc?

Given that memory issues tend to be most visible after a given
process has been running for a long time, it would be great to have
the option to enable heap tracking and profiling at runtime, 
without
having to restart the process. (This ability could then be 
connected
to a Mesos endpoint, similar to how we can adjust the log level at
runtime)

The two production-quality memory allocators that have this ability
currently seem to be tcmalloc and jemalloc. Of these, jemalloc does
produce in our experience better and more detailed statistics.


# What is the impact on users who do not need this feature?

Naturally, not every single user of Mesos will have a need for this
feature. To ensure these users would not experience serious
performance regressions as a result of this change, we conducted a
preliminary set of benchmarks whose results are collected under
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fiss
ues.apache.org%2Fjira%2Fbrowse%2FMESOS-7876&data=02%7C01%7CJeff.Coff
ler%40microsoft.com%7Ccb0bfb1eb3e242c0dd4108d4e946d709%7C72f988bf86f
141af91ab2d7cd011db47%7C1%7C0%7C636389937852266742&sdata=RsZcAGuFm%2
Bw2PPLgMql%2B9vVgkFQrZZFJYdPGcBODsCU%3D&reserved=0

It turns out that we could probably even expect a small speedup (1%
- 5%) as a nice side-effect of this change.

Users who compile Mesos themselves would of course have the option
to disable jemalloc at configuration time or replace it with their
memory allocator of choice.



I'm looking forward to hear any thoughts and comments.


Thanks,
--
Benno Evers
Software Engineer, Mesosphere




--
Benno Evers
Software Engineer, Mesosphere




--
Benno Evers
Software Engineer, Mesosphere