Todd Lipcon has submitted this change and it was merged. ( )

Change subject: Make tcmalloc heap sampling more useful

Make tcmalloc heap sampling more useful

Previously our /pprof/heap path required that the process be started
with lifetime heap profiling enabled. This mode of operation is very
slow so we would never be able to use it in production. Even in tests
it's of limited utility.

The alternative is tcmalloc heap _sampling_ which only records periodic
allocations. According to recent mailing list threads, Google runs most
of their binaries with this enabled in production, so it's designed to
be very low overhead and potentially useful for us in the field.

This doesn't go so far as to enable it by default, but it does adjust
the /pprof/heap endpoint to dump the sampled allocations instead of
starting a profile. Sampling can be enabled using either the tcmalloc
env variable (TCMALLOC_SAMPLE_PARAMETER) or a new command line flag

I tested this manually by running a kudu tserver with loadgen and
sampling set to 500kb. I then used pprof to grab the heap sample
information which looked like I expected it to (most of the memory from

Change-Id: I939c2f01f17ceb4b9520bb566f66463952b2a255
Tested-by: Kudu Jenkins
Reviewed-by: Will Berkeley <>
M docs/troubleshooting.adoc
M src/kudu/server/
M src/kudu/util/
3 files changed, 95 insertions(+), 26 deletions(-)

  Kudu Jenkins: Verified
  Will Berkeley: Looks good to me, approved

To view, visit
To unsubscribe, visit

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I939c2f01f17ceb4b9520bb566f66463952b2a255
Gerrit-Change-Number: 9263
Gerrit-PatchSet: 4
Gerrit-Owner: Todd Lipcon <>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <>
Gerrit-Reviewer: Will Berkeley <>

Reply via email to