Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24401
Change subject: IMPALA-14702 (prep): Add an abstraction around malloc implementation details ...................................................................... IMPALA-14702 (prep): Add an abstraction around malloc implementation details This adds a MallocUtil abstraction around the malloc implementation to make it easier to try other mallocs. This has different MallocUtil libraries for different malloc implementations: - gperftools tcmalloc - sanitizer builds (which have their own malloc implementation) - libc malloc (used only for certain tests). Programs should call MallocUtil::Allocate() early in startup, set any relevant startup flags, then call MallocUtil::Init() to apply the settings. This introduces a tcmalloc_aggressive_memory_decommit startup flag, but limits the configurability and retain existing behavior. Impalad is hard coded to use tcmalloc_aggressive_memory_decommit=true and does not respect a user's value. Other binaries default to tcmalloc_aggressive_memory_decommit=false but will respect a custom value. Impalad's memory tracker has an entry for the malloc overhead. This converts it to a regular metric so that it is available for other daemons. Testing: - Ran core job - Ran pprof locally against the heap profiler endpoint Change-Id: I3ac7841cf279ac40141147d4be9192894d6dc908 --- M be/CMakeLists.txt M be/src/common/daemon-env.cc M be/src/common/global-flags.cc M be/src/common/init.cc M be/src/runtime/bufferpool/system-allocator.cc M be/src/runtime/data-stream-test.cc M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/runtime/mem-tracker.cc M be/src/runtime/query-exec-mgr.cc M be/src/runtime/test-env.cc M be/src/scheduling/admission-controller.cc M be/src/service/impala-hs2-server.cc M be/src/util/CMakeLists.txt A be/src/util/malloc-util-gperftools.h A be/src/util/malloc-util-libc.h A be/src/util/malloc-util-sanitizers.h A be/src/util/malloc-util.cc A be/src/util/malloc-util.h M be/src/util/memory-metrics.cc M be/src/util/memory-metrics.h M be/src/util/memusage-path-handlers.cc M be/src/util/pprof-path-handlers.cc M common/thrift/metrics.json 24 files changed, 817 insertions(+), 322 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/24401/1 -- To view, visit http://gerrit.cloudera.org:8080/24401 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I3ac7841cf279ac40141147d4be9192894d6dc908 Gerrit-Change-Number: 24401 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell <[email protected]>
