[
https://issues.apache.org/jira/browse/HADOOP-7154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14244166#comment-14244166
]
Ben Roling commented on HADOOP-7154:
------------------------------------
Thanks for the confirmation Todd!
> Should set MALLOC_ARENA_MAX in hadoop-config.sh
> -----------------------------------------------
>
> Key: HADOOP-7154
> URL: https://issues.apache.org/jira/browse/HADOOP-7154
> Project: Hadoop Common
> Issue Type: Improvement
> Components: scripts
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 1.0.4, 0.22.0
>
> Attachments: hadoop-7154.txt
>
>
> New versions of glibc present in RHEL6 include a new arena allocator design.
> In several clusters we've seen this new allocator cause huge amounts of
> virtual memory to be used, since when multiple threads perform allocations,
> they each get their own memory arena. On a 64-bit system, these arenas are
> 64M mappings, and the maximum number of arenas is 8 times the number of
> cores. We've observed a DN process using 14GB of vmem for only 300M of
> resident set. This causes all kinds of nasty issues for obvious reasons.
> Setting MALLOC_ARENA_MAX to a low number will restrict the number of memory
> arenas and bound the virtual memory, with no noticeable downside in
> performance - we've been recommending MALLOC_ARENA_MAX=4. We should set this
> in hadoop-env.sh to avoid this issue as RHEL6 becomes more and more common.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
