[
https://issues.apache.org/jira/browse/HADOOP-7154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14230532#comment-14230532
]
Ben Roling commented on HADOOP-7154:
------------------------------------
[~tlipcon] - I know this bug is pretty old, but do you mind doing me the favor
of explaining this statement:
{quote}
We've observed a DN process using 14GB of vmem for only 300M of resident set.
This causes all kinds of nasty issues for obvious reasons.
{quote}
Perhaps I am revealing too much of my naivety, but what issues the vmem size
presents nor the reasons are necessarily obvious to me. The reason I ask is
not directly related to this JIRA nor even Hadoop. I am just trying to learn
more about the glibc change and its potential impacts. I've noticed high
virtual memory size in another Java-based application (a Zabbix agent process
if you care) and I'm struggling slightly to decide if I should worry about it.
http://journal.siddhesh.in/posts/malloc-per-thread-arenas-in-glibc.html
presents what appears to me to be a rational explanation as to why the virtual
memory size shouldn't matter too much.
I could push on Zabbix to implement a change to set MALLOC_ARENA_MAX and I feel
relatively confident the change wouldn't hurt anything but I'm not sure it
would actually help anything either. The Zabbix agent appears to be performing
fine and the only reason I noticed the high vmem size was because someone
pointed me to this JIRA and I did an audit looking for processes with virtual
memory sizes that looked suspicious.
I guess the biggest problem I have with the affect the glibc change has on
reported vmem size is that it seems to make vmem size meaningless where
previously you could get some idea about what a process was doing from its vmem
size but your comment suggests maybe there are other things I should be
concerned about as well. If you could share those with me I would greatly
appreciate it and perhaps others will benefit as well.
Thanks!
> Should set MALLOC_ARENA_MAX in hadoop-config.sh
> -----------------------------------------------
>
> Key: HADOOP-7154
> URL: https://issues.apache.org/jira/browse/HADOOP-7154
> Project: Hadoop Common
> Issue Type: Improvement
> Components: scripts
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 1.0.4, 0.22.0
>
> Attachments: hadoop-7154.txt
>
>
> New versions of glibc present in RHEL6 include a new arena allocator design.
> In several clusters we've seen this new allocator cause huge amounts of
> virtual memory to be used, since when multiple threads perform allocations,
> they each get their own memory arena. On a 64-bit system, these arenas are
> 64M mappings, and the maximum number of arenas is 8 times the number of
> cores. We've observed a DN process using 14GB of vmem for only 300M of
> resident set. This causes all kinds of nasty issues for obvious reasons.
> Setting MALLOC_ARENA_MAX to a low number will restrict the number of memory
> arenas and bound the virtual memory, with no noticeable downside in
> performance - we've been recommending MALLOC_ARENA_MAX=4. We should set this
> in hadoop-env.sh to avoid this issue as RHEL6 becomes more and more common.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
