[
https://issues.apache.org/jira/browse/IMPALA-14702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18089506#comment-18089506
]
ASF subversion and git services commented on IMPALA-14702:
----------------------------------------------------------
Commit 0b8294b1a5b0ad4a817dee13b7fbb2ee53f534e2 in impala's branch
refs/heads/master from Joe McDonnell
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=0b8294b1a ]
IMPALA-14702 (prep): Add an abstraction around malloc implementation details
This adds a MallocUtil abstraction around the malloc implementation
to make it easier to try other mallocs. This has different
MallocUtil libraries for different malloc implementations:
- gperftools tcmalloc
- sanitizer builds (which have their own malloc implementation)
- libc malloc (used only for certain tests).
Programs should set any relevant startup flags, then call
MallocUtil::Init() to apply the settings.
This introduces a tcmalloc_aggressive_memory_decommit startup
flag, but limits the configurability and retain existing behavior.
Impalad is hard coded to use tcmalloc_aggressive_memory_decommit=true
and does not respect a user's value. Other binaries default to
tcmalloc_aggressive_memory_decommit=false but will respect a
custom value.
Impalad's memory tracker has an entry for the malloc overhead.
This converts it to a regular metric so that it is available
for other daemons.
Testing:
- Ran core job
- Ran pprof locally against the heap profiler endpoint
Change-Id: I3ac7841cf279ac40141147d4be9192894d6dc908
Reviewed-on: http://gerrit.cloudera.org:8080/24401
Reviewed-by: Joe McDonnell <[email protected]>
Reviewed-by: Michael Smith <[email protected]>
Tested-by: Joe McDonnell <[email protected]>
> Add support for building against other malloc implementations (e.g. jemalloc)
> -----------------------------------------------------------------------------
>
> Key: IMPALA-14702
> URL: https://issues.apache.org/jira/browse/IMPALA-14702
> Project: IMPALA
> Issue Type: Task
> Components: Backend
> Affects Versions: Impala 5.0.0
> Reporter: Joe McDonnell
> Priority: Major
>
> Impala builds against gperftools tcmalloc. Other malloc implementations like
> jemalloc and Google's tcmalloc have different designs that would be
> interesting to test. We know that there are performance improvements related
> to avoiding gperftool tcmalloc's aggressive decommit setting. It would be
> nice to test that against other malloc implementations (or forgo overriding
> malloc altogether).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
